Multi-touch contact tracking using predicted paths

ABSTRACT

Apparatus and methods are disclosed for simultaneously tracking multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing, multi-touch surface. Identification and classification of intuitive hand configurations and motions enables unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and handwriting into a versatile, ergonomic computer input device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 11/015,434,(now U.S. Pat. No. 7,339,580), entitled Contact Tracking andIdentification Module for Touch Sensing, filed Dec. 17, 2004; which is acontinuation of application Ser. No. 09/919,266 (now U.S. Pat. No.6,888,536), entitled “Method And Apparatus For Integrating Manual Input”filed Jul. 31, 2001; which is a division of application Ser. No.09/236,513 (now U.S. Pat. No. 6,323,846) filed Jan. 25, 1999 whichclaims the benefit of provisional Application No. 60/072,509, filed Jan.26, 1998, each of which is hereby incorporated by reference in itsentirety. This application is also related to application Ser. No.11/428,501 entitled “Capacitive Sensing Arrangement” filed Jul. 3, 2006;application Ser. No. 11/428,503 entitled “Touch Surface” filed Jul. 3,2006; application Ser. No. 11/428,506 entitled “User Interface Gestures”filed Jul. 3, 2006; application Ser. No. 11/428,515 entitled “UserInterface Gestures” filed Jul. 3, 2006, application Ser. No. 11/428,522entitled “Identifying Contacts on a Touch Surface” filed Jul. 3, 2006;application Ser. No. 11/428,521 (now U.S. Pat. No. 7,619,619), entitled“Identifying Contacts on a Touch Surface” filed Nov. 14, 2006;application Ser. No. 11/559,763, entitled “Multi-Touch Contact MotionExtraction” filed Nov. 14, 2006; application Ser. No. 11/559,799,entitled “Multi-Touch Contact Motion Extraction”; filed Nov. 14, 2006and application Ser. No. 11/559,833, entitled “Multi-Touch Hand PositionOffset Computation”; filed Nov. 14, 2006, each of which is herebyincorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

A. Field of the Invention

The present invention relates generally to methods and apparatus fordata input, and, more particularly, to a method and apparatus forintegrating manual input.

B. Description of the Related Art

Many methods for manual input of data and commands to computers are inuse today, but each is most efficient and easy to use for particulartypes of data input. For example, drawing tablets with pens or pucksexcel at drafting, sketching, and quick command gestures. Handwritingwith a stylus is convenient for filling out forms which requiresignatures, special symbols, or small amounts of text, but handwritingis slow compared to typing and voice input for long documents. Mice,finger-sticks and touchpads excel at cursor pointing and graphicalobject manipulations such as drag and drop. Rollers, thumbwheels andtrackballs excel at panning and scrolling. The diversity of tasks thatmany computer users encounter in a single day call for all of thesetechniques, but few users will pay for a multitude of input devices, andthe separate devices are often incompatible in a usability and anergonomic sense. For instance, drawing tablets are a must for graphicsprofessionals, but switching between drawing and typing is inconvenientbecause the pen must be put down or held awkwardly between the fingerswhile typing. Thus, there is a long-felt need in the art for a manualinput device which is cheap yet offers convenient integration of commonmanual input techniques.

Speech recognition is an exciting new technology which promises torelieve some of the input burden on user hands. However, voice is notappropriate for inputting all types of data either. Currently, voiceinput is best-suited for dictation of long text documents. Until naturallanguage recognition matures sufficiently that very high level voicecommands can be understood by the computer, voice will have littleadvantage over keyboard hot-keys and mouse menus for command andcontrol. Furthermore, precise pointing, drawing, and manipulation ofgraphical objects is difficult with voice commands, no matter how wellspeech is understood. Thus, there will always be a need in the art formulti-function manual input devices which supplement voice input.

A generic manual input device which combines the typing, pointing,scrolling, and handwriting capabilities of the standard input devicecollection must have ergonomic, economic, and productivity advantageswhich outweigh the unavoidable sacrifices of abandoning devicespecialization. The generic device must tightly integrate yet clearlydistinguish the different types of input. It should therefore appearmodeless to the user in the sense that the user should not need toprovide explicit mode switch signals such as buttonpresses, armrelocations, or stylus pickups before switching from one input activityto another. Epidemiological studies suggest that repetition and forcemultiply in causing repetitive strain injuries. Awkward postures, deviceactivation force, wasted motion, and repetition should be minimized toimprove ergonomics. Furthermore, the workload should be spread evenlyover all available muscle groups to avoid repetitive strain.

Repetition can be minimized by allocating to several graphicalmanipulation channels those tasks which require complex mouse pointermotion sequences. Common graphical user interface operations such asfinding and manipulating a scroll bar or slider control are much lessefficient than specialized finger motions which cause scrollingdirectly, without the step of repositioning the cursor over an on-screencontrol. Preferably the graphical manipulation channels should bedistributed amongst many finger and hand motion combinations to spreadthe workload. Touchpads and mice with auxilliary scrolling controls suchas the Cirque®™ Smartcat touchpad with edge scrolling, the IBM®™ScrollPoint™ mouse with embedded pointing stick, and the Roller Mousedescribed in U.S. Pat. No. 5,530,455 to Gillick et al. represent smallimprovements in this area, but still do not provide enough directmanipulation channels to eliminate many often-used cursor motionsequences. Furthermore, as S. Zhai et al. found in “Dual Stream Inputfor Pointing and Scrolling,” Proceedings of CHI '97 Extended Abstracts(1997), manipulation of more than two degrees of freedom at a time isvery difficult with these devices, preventing simultaneous panning,zooming and rotating.

Another common method for reducing excess motion and repetition is toautomatically continue pointing or scrolling movement signals once theuser has stopped moving or lifts the finger. Related art methods can bedistinguished by the conditions under which such motion continuation isenabled. In U.S. Pat. No. 4,734,685, Watanabe continues image panningwhen the distance and velocity of pointing device movement exceedthresholds. Automatic panning is, stopped by moving the pointing deviceback in the opposite direction, so stopping requires additional precisemovements. In U.S. Pat. No. 5,543,591 to Gillespie et al., motioncontinuation occurs when the finger enters an edge border region arounda small touchpad. Continued motion speed is fixed and the directioncorresponds to the direction from the center of the touchpad to thefinger at the edge. Continuation mode ends when the finger leaves theborder region or lifts off the pad. Disadvantageously, users sometimespause at the edge of the pad without intending for cursor motion tocontinue, and the unexpected motion continuation becomes annoying. U.S.Pat. No. 5,327,161 to Logan et al. describes motion continuation whenthe finger enters a border area as well, but in an alternative trackballemulation mode, motion continuation can be a function solely of lateralfinger velocity and direction at liftoff. Motion continuation decays dueto a friction factor or can be stopped by a subsequent touchdown on thesurface. Disadvantageously, touch velocity at liftoff is not a reliableindicator of the user's desire for motion continuation since whenapproaching a large target on a display at high speeds the user may notstop the pointer completely before liftoff. Thus it would be an advancein the art to provide a motion continuation method which does not becomeactivated unexpectedly when the user really intended to stop pointermovement at a target but happens to be on a border or happens to bemoving at significant speed during liftoff.

Many attempts have been made to embed pointing devices in a keyboard sothe hands do not have to leave typing position to access the pointingdevice. These include the integrated pointing key described in U.S. Pat.No. 5,189,403 to Franz et al., the integrated pointing stick disclosedby J. Rutledge and T. Selker in “Force-to-Motion Functions forPointing,” Human-Computer Interaction—INTERACT '90, pp. 701-06 (1990),and the position sensing keys described in U.S. Pat. No. 5,675,361 toSantilli. Nevertheless, the limited movement range and resolution ofthese devices, leads to poorer pointing speed and accuracy than a mouse,and they add mechanical complexity to keyboard construction. Thus thereexists a need in the art for pointing methods with higher resolution,larger movement range, and more degrees of freedom yet which are easilyaccessible from typing hand positions.

Touch screens and touchpads often distinguish pointing motions fromemulated button clicks or keypresses by assuming very little lateralfingertip motion will occur during taps on the touch surface which areintended as clicks. Inherent in these methods is the assumption thattapping will usually be straight down from the suspended fingerposition, minimizing those components of finger motion tangential to thesurface. This is a valid assumption if the surface is not finely dividedinto distinct key areas or if the user does a slow, “hunt and peck”visual search for each key before striking. For example, in U.S. Pat.No. 5,543,591 to Gillespie et al., a touchpad sends all lateral motionsto the host computer as cursor movements. However, if the finger islifted soon enough after touchdown to count as a tap and if theaccumulated lateral motions are not excessive, any sent motions areundone and a mouse button click is sent instead. This method only worksfor mouse commands such as pointing which can safely be undone, not fordragging or other manipulations. In U.S. Pat. No. 5,666,113 to Logan,taps with less than about 1/16″ lateral motion activate keys on a smallkeypad while lateral motion in excess of 1/16″ activates cursor controlmode. In both patents cursor mode is invoked by default when a fingerstays on the surface a long time.

However, fast touch typing on a surface divided into a large array ofkey regions tends to produce more tangential motions along the surfacethan related art filtering techniques can tolerate. Such an arraycontains keys in multiple rows and columns which may not be directlyunder the fingers, so the user must reach with the hand or flex orextend fingers to touch many of the key regions. Quick reaching andextending imparts significant lateral finger motion while the finger isin the air which may still be present when the finger contacts thesurface. Glancing taps with as much as ¼″) lateral motion measured atthe surface can easily result. Attempting to filter or suppress thismuch motion would make the cursor seem sluggish and unresponsive.Furthermore, it may be desirable to enter a typematic or automatic keyrepeat mode instead of pointing mode when the finger is held in oneplace on the surface. Any lateral shifting by the fingertip during aprolonged finger press would also be picked up as cursor jitter withoutheavy filtering. Thus, there is a need in the art for a method todistinguish keying from pointing on the same surface via more robusthand configuration cues than lateral motion of a single finger.

An ergonomic typing system should require minimal key tapping force,easily distinguish finger taps from resting hands, and cushion thefingers from the jarring force of surface impact. Mechanical andmembrane keyboards rely on the spring force in the keyswitches toprevent activation when the hands are resting on the keys. This causesan irreconcilable tradeoff between the ergonomic desires to reduce thefatigue from key activating force and to relax the full weight of thehands onto the keys during rest periods. Force minimization on touchsurfaces is possible with capacitive or active optical sensing, which donot rely on finger pressure, rather than resistive-membrane orsurface-acoustic-wave sensing techniques. The related art touch devicesdiscussed below will become confused if a whole hand including its fourfingertips a thumb and possibly palm heels, rests on the surface. Thus,there exists a long felt need in the art for a multi-touch surfacetyping system based on zero-force capacitive sensing which can tolerateresting hands and a surface cushion.

An ergonomic typing system should also adapt to individual hand sizestolerate variations in typing style, and support a range of healthy handpostures. Though many ergonomic keyboards have been proposed, mechanicalkeyswitches can only be repositioned at great cost. For example, thekeyboard with concave keywells described by Hargreaves et al. in U.S.Pat. No. 5,689,253 fits most hands well but also tends to lock the armsin a single position. A touch surface key layout could easily bemorphed, translated, or arbitrarily reconfigured as long as the changesdid not confuse the user. However, touch surfaces may not provide asmuch laterally orienting tactile feedback as the edges of mechanicalkeyswitches. Thus, there exists a need in the art for a surface typingrecognizer which can adapt a key layout to fit individual hand posturesand which can sustain typing accuracy if the hands drift due to limitedtactile feedback.

Handwriting on smooth touch surfaces using a stylus is well-known in theart, but it typically does not integrate well with typing and pointingbecause the stylus must be put down somewhere or held awkwardly duringother input activities. Also, it may be difficult to distinguish thehandwriting activity of the stylus from pointing motions of a fingertip.Thus there exists a need in the art for a method to capture coarsehandwriting gestures without a stylus and without confusing them withpointing motions.

Many of the input differentiation needs cited above could be met with atouch sensing technology which distinguishes a variety of handconfigurations and motions such as sliding finger chords and grips. Manymechanical chord keyboards have been designed to detect simultaneousdownward activity from multiple fingers, but they do not detect lateralfinger motion over a large range. Related art shows several examples ofcapacitive touchpads which emulate a mouse or keyboard by tracking asingle finger. These typically measure the capacitance of or betweenelongated wires which are laid out in rows and columns. A thindielectric is interposed between the row and column layers. Presence ofa finger perturbs the self or mutual capacitance for nearby electrodes.Since most of these technologies use projective row and column sensorswhich integrate on one electrode the proximity of all objects in aparticular row or column, they cannot uniquely determine the positionsof two or more objects as discussed in S. Lee, “A FastMultiple-Touch-Sensitive Input Device,” University of Toronto MastersThesis (1984). The best they can do is count fingertips which happen tolie in a straight row, and even that will fail if a thumb or palm isintroduced in the same column as a fingertip.

In U.S. Pat. Nos. 5,565,658 and 5,305,017, Gerpheide et al. measure themutual capacitance between row and column electrodes by driving one setof electrodes at some clock frequency and sensing how much of thatfrequency is coupled onto a second electrode set. Such synchronousmeasurements are very prone to noise at the driving frequency, so toincrease signal-to-noise ratio they form virtual electrodes comprised ofmultiple rows or multiple columns, instead of a single row and column,and scan through electrode combinations until the various mutualcapacitances are nulled or balanced. The coupled signal increases withthe product of the rows and columns in each virtual electrodes, but thenoise only increases with the sum, giving a net gain in signal-to-noiseratio for virtual electrodes consisting of more than two rows and twocolumns. However, to uniquely distinguish multiple objects, virtualelectrode sizes would have to be reduced so the intersection of the rowand column virtual electrodes would be no larger than a finger tip,i.e., about two rows and two columns, which will degrade thesignal-to-noise ratio. Also, the signal-to-noise ratio drops as row andcolumn lengths increase to cover a large area.

In U.S. Pat. Nos. 5,543,591, 5,543,590, and 5,495,077, Gillespie et almeasure the electrode-finger self-capacitance for row and columnelectrodes independently. Total electrode capacitance is estimated bymeasuring the electrode voltage change caused by injecting or removing aknown amount of charge in a known time. All electrodes can be measuredsimultaneously if each electrode has its own drive/sense circuit. Thecentroid calculated from all row and column electrode signalsestablishes an interpolated vertical and horizontal position for asingle object. This method may in general have higher signal-to-noiseratio than synchronous methods, but the signal-to-noise ratio is stilldegraded as row and column lengths increase. Signal-to-noise ratio isespecially important for accurately locating objects which are floatinga few millimeters above the pad. Though this method can detect suchobjects, it tends to report their position as being near the middle ofthe pad, or simply does not detect floating objects near the edges.

Thus there exists a need in the art for a capacitance-sensing apparatuswhich does not suffer from poor signal-to-noise ratio and the multiplefinger indistinguishability problems of touchpads with long row andcolumn electrodes.

U.S. Pat. No. 5,463,388 to Boie et al. has a capacitive sensing systemapplicable to either keyboard or mouse input, but does not consider theproblem of integrating both types of input simultaneously. Though theymention independent detection of arrayed unit-cell electrodes, theircapacitance transduction circuitry appears too complex to beeconomically reproduced at each electrode. Thus the long lead wiresconnecting electrodes to remote signal conditioning circuitry can pickupnoise and will have significant capacitance compared to thefinger-electrode self-capacitance, again limiting signal-to-noise ratio.Also, they do not recognize the importance of independent electrodes formultiple finger tracking, or mention how to track multiple fingers on anindependent electrode array.

Lee built an early multi-touch electrode array, with 7 mm by 4 mm metalelectrodes arranged in 32 rows and 64 columns. The “FastMultiple-Touch-Sensitive Input Device (FMTSID)” total active areameasured 12″ by 16″, with a 0.075 mm Mylar dielectric to insulatefingers from electrodes. Each electrode had one diode connected to a rowcharging line and a second diode connected to a column discharging line.Electrode capacitance changes were measured singly or in rectangulargroups by raising the voltage on one or more row lines, selectivelycharging the electrodes in those rows, and then timing the discharge ofselected columns to ground through a discharge resistor. Lee's designrequired only two diodes per electrode, but the principal disadvantageof Lee's design is that the column diode reverse bias capacitancesallowed interference between electrodes in the same column.

All of the related capacitance sensing art cited above utilizeinterpolation between electrodes to achieve high pointing resolutionwith economical electrode density. Both Boie et al. and Gillespie et al.discuss compultattion of a centroid from all row and column electrodereadings. However, for multiple finger detection, centroid calculationmust be carefully limited around local maxima to include only one fingerat a time. Lee utilizes a bisective search technique to find localmaxima and then interpolates only on the eight nearest neighborelectrodes of each local maximum electrode. This may work fine for smallfingertips, but thumb and palm contacts may cover more than nineelectrodes. Thus there exists a need in the art for improved means togroup exactly those electrodes which are covered by each distinguishablehand contact and to compute a centroid from such potentially irregulargroups.

To take maximum advantage of multi-touch surface sensing, complexproximity image processing is necessary to track and identify the partsof the hand contacting the surface at any one time. Compared to passiveoptical, images, proximity images provide clear indications of where thebody contacts the surface, uncluttered by luminosity variation andextraneous objects in the background. Thus proximity image filtering andsegmentation stages can be simpler and more reliable than in computervision approaches to free-space hand tracking such as S. Alimad, “AUsable Real-Time 3D Hand Tracker,” Proceedings of the 28^(th) AsilomarConference on Signals, Systems, and Computers—Part 2, vol. 2, IEEE(1994) or Y. Cui and J. Wang, “Hand Segmentation Using Learning-BasedPrediction and Verification for Hand Sign Recognition,” Proceedings ofthe 1996 IEEE Computer Society Conference on Computer Vision and PatternRecognition, pp. 88-93 (1996). However, parts of the hand such asintermediate finger joints and the center of the palms do not show up incapacitive proximity images at all if the hand is not flattened on thesurface. Without these intermediate linkages between fingertips andpalms the overall hand structure can only be guessed at, making handcontact identification very difficult. Hence the optical flow andcontour tracking techniques which have been applied to free-space handsign language recognition as in F. Quek, “Unencumbered GesturalInteraction,” IEEE Multimedia, vol. 3, pp. 36-47 (1996), do not addressthe special challenges of proximity image tracking.

Synaptics Corp. has successfully fabricated their electrode array onflexible mylar film rather than stiff circuit board. This is suitablefor conforming to the contours of special products, but does not providesignificant finger cushioning for large surfaces. Even if a cushion wasplaced under the film, the lack of stretchability in the film, leads,and electrodes would limit the compliance afforded by the compressiblematerial. Boie et al suggests that placing cornnressihle insulators ontop of the electrode array cushions finger impact. However, an insulatormore than about one millimeter thick would seriously attenuate themeasured finger-electrode capacitances. Thus there exists a need in theart for a method to transfer finger capacitance influences through anarbitrarily thick cushion.

SUMMARY OF THE INVENTION

It is a primary object of the present invention to provide a system andmethod for integrating different types of manual input such as typing,multiple degree-of-freedom manipulation, and handwriting on amulti-touch surface.

It is also an object of the present invention to provide a system andmethod for distinguishing different types of manual input such astyping, multiple degree-of-freedom manipulation, and handwriting on amulti-touch surface, via different hand configurations which are easyfor the user to learn and easy for the system to recognize.

It is a further object of the present invention to provide an improvedcapacitance-transducing apparatus that is cheaply implemented near eachelectrode so that two-dimensional sensor arrays of arbitrary size andresolution can be built without degradation in signal to noise.

It is a further object of the present invention to provide an electronicsystem which minimizes the number of sensing electrodes necessary toobtain proximity images with such resolution that a variety of handconfigurations can be distinguished.

Yet another object of the present invention is to provide a multi-touchsurface apparatus which is compliant and contoured to be comfortable andergonomic under extended use.

Yet another object of the present invention is to provide tactile key orhand position feedback without impeding hand resting on the surface orsmooth, accurate sliding across the surface.

It is a further object of the present invention to provide an electronicsystem which can provide images of flesh proximity to an array ofsensors with such resolution that a variety of hand configurations canbe distinguished.

It is another object of the present invention to provide an improvedmethod for invoking cursor motion continuation only when the user wantsit by not invoking it when significant deceleration is detected.

Another object of the present invention is to identify different handparts as they contact the surface so that a variety of handconfigurations can be recognized and used to distinguish different kindsof input activity.

Yet another object of the present invention is to reliably extractrotation and scaling as well as translation degrees of freedom from themotion of two or more hand contacts to aid in navigation andmanipulation of two-dimensional electronic documents.

It is a further object of the present invention to reliably extract tiltand roll degrees of freedom from hand pressure differences to aid innavigation and manipulation of three-dimensional environments.

Additional objects and advantages of the invention will be set forth inpart in the description which follows, and in part will be obvious fromthe description, or may be learned by practice of the invention. Theobjects and advantages of the invention will be realized and attained bymeans of the elements and combinations particularly pointed out in theappended claims.

To achieve the objects and in accordance with the purpose of theinvention, as embodied and broadly described herein, the inventioncomprises a sensing device that is sensitive to changes inself-capacitance brought about by changes in proximity of a touch deviceto the sensing device, the sensing device comprising: two electricalswitching means connected together in series having a common node, aninput node, and an output node; a dielectric-covered sensing electrodeconnected to the common node between the two switching means; a powersupply providing an approximately constant voltage connected to theinput node of the series-connected switching means; an integratingcapacitor to accumulate charge transferred during multiple consecutiveswitchings of the series connected switching means; another switchingmeans connected in parallel across the integrating capacitor to depleteits residual charge; and a voltage-to-voltage translation deviceconnected to the output node of the series-connected switching meanswhich produces a voltage representing the magnitude of theself-capacitance of the sensing device. Alternatively, the sensingdevice comprises: two electrical switching means connected together inseries having a common node, an input node, and an output node; adielectric-covered sensing electrode connected to the common nodebetween the two switching means; a power supply providing anapproximately constant voltage connected to the input node of theseries-connected switching means; and an integrating current-to-voltagetranslation device connected to the output node of the series connectedswitching means, the current-to-voltage translation device producing avoltage representing the magnitude of the self-capacitance of thesensing device.

To further achieve the objects, the present invention comprises amulti-touch surface apparatus for detecting a spatial arrangement ofmultiple touch devices on or near the surface of the multi-touchapparatus, comprising: one of a rigid or flexible surface; a pluralityof two-dimensional arrays of one of the sensing devices (recited in theprevious paragraph) arranged on the surface in groups wherein thesensing devices within a group have their output nodes connectedtogether and share the same integrating capacitor, charge depletionswitch, and voltage-to-voltage translation circuitry; control circuitryfor enabling a single sensor device from each two-dimensional array;means for selecting the sensor voltage data from each two-dimensionalarray; voltage measurement circuitry to convert sensor voltage data to adigital code; and circuitry for communicating the digital code toanother electronic device. The sensor voltage data selecting meanscomprises one of a multiplexing circuitry and a plurality of voltagemeasurement circuits.

To still further achieve the objects, the present invention comprises amulti-touch surface apparatus for sensing diverse configurations andactivities of touch devices and generating integrated manual input toone of an electronic or electromechanical device, the apparatuscomprising: an array of one of the proximity sensing devices describedabove; a dielectric cover having symbols printed thereon that representaction-to-be-taken when engaged by the touch devices; scanning means forforming digital proximity images from the array of sensing devices;calibrating means for removing background offsets from the proximityimages; recognition means for interpreting the configurations andactivities of the touch devices that make up the proximity images;processing means for generating input signals in response to particulartouch device configurations and motions; and communication means forsending the input signals to the electronic or electromechanical device.

To even further achieve the objects, the present invention comprises amulti-touch surface apparatus for sensing diverse configurations andactivities of fingers and palms of one or more hands near the surfaceand generating integrated manual input to one of an electronic orelectromechanical device, the apparatus comprising: an array ofproximity sensing means embedded in the surface; scanning means forforming digital proximity images from the proximities measured by thesensing means; image segmentation means for collecting into groups thoseproximity image pixels intensified by contact of the samedistinguishable part of a hand; contact tracking means forparameterizing hand contact features and trajectories as the contactsmove across successive proximity images, contact identification meansfor determining which hand and which part of the hand is causing eachsurface contact; synchronization detection means for identifying subsetsof identified contacts which touchdown or liftoff the surface atapproximately the same time, and for generating command signals inresponse to synchronous taps of multiple fingers on the surface; typingrecognition means for generating intended key symbols from asynchronousfinger taps; motion component extraction means for compressing multipledegrees of freedom of multiple fingers into degrees of freedom common intwo and three dimensional graphical manipulation; chord motionrecognition means for generating one of command and cursor manipulationsignals in response to motion in one or more extracted degrees offreedom by a selected combination of fingers; pen grip detection meansfor recognizing contact arrangements which resemble the configuration ofthe hand when gripping a pen, generating inking signals from motions ofthe inner fingers, and generating cursor manipulation signals frommotions of the palms while the inner fingers are lifted; andcommunication means for sending the sensed configurations and activitiesof finger and palms to one of the electronic and electromechanicaldevice.

To further achieve the objects, the present invention comprises a methodfor tracking and identifying hand contacts in a sequence of proximityimages in order to support interpretation of hand configurations andactivities related to typing, multiple degree-of-freedom manipulationvia chords, and handwriting, the method comprising the steps of:segmenting each proximity image into groups of electrodes which indicatesignificant proximity, each group representing proximity of adistinguishable hand part or other touch device; extracting totalproximity, position, shape, size, and orientation parameters from eachgroup of electrodes; tracking group paths through successive proximityimages including detection of path endpoints at contact touchdown andliftoff; computing velocity and filtered position vectors along eachpath; assigning a hand and finger identity to each contact path byincorporating relative path positions and velocities, individual contactfeatures, and previous estimates of hand and finger positions; andmaintaining estimates of hand and finger positions from trajectories ofpaths currently assigned to the fingers, wherein the estimates providehigh level feedback to bias segmentations and identifications in futureimages.

To still further achieve the objects, the present invention comprises amethod for integrally extracting multiple degrees of freedom of handmotion from sliding motions of two or more fingers of a hand across amulti-touch surface, one of the fingers preferably being the opposablethumb, the method comprising the steps of: tracking across successivescans of the proximity sensor array the trajectories of individual handparts on the surface; finding an innermost and an outermost fingercontact from contacts identified as fingers on the given hand; computinga scaling velocity component from a change in a distance between theinnermost and outermost finger contacts; computing a rotational velocitycomponent from a change in a vector angle between the innermost andoutermost finger contacts; computing a translation weighting for eachcontacting finger; computing translational velocity components in twodimensions from a translation weighted average of the finger velocitiestangential to surface; suppressively filtering components whose speedsare consistently lower than the fastest components; transmitting thefiltered velocity components as control signals to an electronic orelectromechanical device.

To even further achieve the objects, the present invention comprises amanual input integration method for supporting diverse hand inputactivities such as resting the hands, typing, multiple degree-of-freedommanipulation, command gesturing and handwriting on a multi-touchsurface, the method enabling users to instantaneously switch between theinput activities by placing their hands in different configurationscomprising distinguishable combinations of relative hand contact timing,proximity, shape, size, position, motion and/or identity across asuccession of surface proximity images, the method comprising the stepsof: tracking each touching hand part across successive proximity images;measuring the times when each hand part touches down and lifts off thesurface; detecting when hand parts touch down or lift offsimultaneously; producing discrete key symbols when the userasynchronously taps, holds, or slides a finger on key regions defined onthe surface; producing discrete mouse button click commands, keycommands, or no signals when the user synchronously taps two or morefingers from the same hand on the surface; producing gesture commands ormultiple degree-of-freedom manipulation signals when the user slides twoor more fingers across the surface; and sending the produced symbols,commands and manipulation signals as input to an electronic or anelectromechanical device.

To still even further achieve the objects, the present inventioncomprises a method for choosing what kinds of input signals will begenerated and sent to an electronic or electromechanical device inresponse to tapping or sliding of fingers on a multi-touch surface, themethod comprising the following steps: identifying each contact on thesurface as either a thumb, fingertip or palm; measuring the times wheneach hand part touches down and lifts off the surface; forming a set ofthose fingers which touch down from the all finger floating state beforeany one of the fingers lifts back off the surface; choosing the kinds ofinput signals to be generated by further distinctive motion of thefingers from the combination of finger identities in the set; generatinginput signals of this kind when further distinctive motions of thefingers occur; forming a subset any two or more fingers which touch downsynchronously after at least one finger has lifted back off the surface;choosing a new kinds of input signals to be generated by furtherdistinctive motion of the fingers from the combination of fingeridentities in the subset; generating input signals of this new kind whenfurther distinctive motions of the fingers occur; and continuing to formnew subsets, choose and generate new kinds of input signals in responseto liftoff and synchronous touchdowns until all fingers lift off thesurface.

To further achieve the objects, the present invention comprises a methodfor continuing generation of cursor movement or scrolling signals from atangential motion of a touch device over a touch-sensitive input devicesurface after touch device liftoff from the surface if the touch deviceoperator indicates that cursor movement continuation is desired byaccelerating or failing to decelerate the tangential motion of the touchdevice before the touch device is lifted, the method comprising thefollowing steps: measuring, storing and transmitting to a computingdevice two or more representative tangential velocities during touchdevice manipulation; computing and storing a liftoff velocity from touchdevice positions immediately prior to the touch device liftoff;comparing the liftoff velocity with the representative tangentialvelocities, and entering a mode for continuously moving the cursor if atangential liftoff direction approximately equals the representativetangential directions and a tangential liftoff speed is greater than apredetermined fractional multiple of representative tangential speeds;continuously transmitting cursor movement signals after liftoff to acomputing device such that the cursor movement velocity corresponds toone of the representative tangential velocities; and ceasingtransmission of the cursor movement signals when the touch deviceengages the surface again, if comparing means detects significantdeceleration before liftoff, or if the computing device replies that thecursor can move no farther or a window can scroll no farther.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and together with the description, serve to explain theprinciples of the invention. In the drawings:

FIG. 1 is a block diagram of the integrated manual input apparatus;

FIG. 2 is a schematic drawing of the proximity sensor with voltageamplifier;

FIG. 3 is a schematic drawing of the proximity sensor with integratingcurrent amplifier;

FIG. 4 is a schematic drawing of the proximity sensor implemented withfield effect transistors;

FIG. 5 is a schematic drawing of the proximity sensor as used toimplement 2D arrays of proximity sensors;

FIG. 6 is a block diagram showing a typical architecture for a 2D arrayof proximity sensors where all sensors share the same amplifier;

FIG. 7 is a block diagram of circuitry used to convert proximity sensoroutput to a digital code;

FIG. 8 is a block diagram showing a typical architecture for a 2D arrayof proximity sensors where sensors within a row share the sameamplifier;

FIG. 9 is a schematic of a circuit useful for enabling the output gatesof all proximity sensors within a group (arranged in columns);

FIG. 10 is a side view of a 2D proximity sensor array that is sensitiveto the pressure exerted by non-conducting touch objects;

FIG. 11 is a, side view of a 2D proximity sensor array that provides acompliant surface without loss of spatial sensitivity;

FIG. 12 is a side view of a 2D proximity sensor array that is sensitiveto both the proximity of conducting touch objects and to the pressureexerted by non-conducting touch objects;

FIG. 13 is an example proximity image of a hand flattened onto thesurface with fingers outstretched;

FIG. 14 is an example proximity image of a hand partially closed withfingertips normal to surface;

FIG. 15 is an example proximity image of a hand in the pen gripconfiguration with thumb and index fingers pinched;

FIG. 16 is a data flow diagram of the hand tracking and contactidentification system;

FIG. 17 is a flow chart of hand position estimation:

FIG. 18 is a data flow diagram of proximity image segmentation;

FIG. 19 is a diagram of the boundary search pattern during constructionof an electrode group;

FIG. 20A is a diagram of the segmentation strictness regions with bothhands in their neutral, default position on surface;

FIG. 20B is a diagram of the segmentation strictness regions when thehands are in asymmetric positions on surface;

FIG. 20C is a diagram of the segmentation strictness regions when theright hand crosses to the left half of the surface and the left hand isoff the surface;

FIG. 21 is a flow chart of segmentation edge testing;

FIG. 22 is a flow chart of persistent path tracking;

FIG. 23 is a flow chart of the hand part identification algorithm;

FIG. 24 is a Voronoi cell diagram constructed around hand part attractorpoints;

FIG. 25A is a plot of orientation weighting factor for right thumb,right inner palm, and left outer palm versus contact orientation;

FIG. 25B is a plot of thumb size factor versus contact size;

FIG. 25C is a plot of palm size factor versus ratio of total contactproximity to contact eccentricity;

FIG. 25D is a plot of palm separation factor versus distance between acontact and it nearest neighbor contact;

FIG. 26 is a flow chart of the thumb presence verification algorithm;

FIG. 27 is a flow chart of an alternative hand part identificationalgorithm;

FIG. 28 is a flow chart of the pen grip detection process:

FIG. 29 is a flow chart of the hand identification algorithm:

FIGS. 30A-C show three different hand partition hypotheses for a fixedarrangement of surface contacts;

FIG. 31A is a plot of the hand clutching direction factor versushorizontal hand velocity;

FIG. 31B is a plot of the handedness factor versus vertical position ofoutermost finger relative to next outermost;

FIG. 31C is a plot of the palm cohesion factor versus maximum horizontalseparation between palm contacts within a hand;

FIG. 32 is a plot of the inner finger angle factor versus the anglebetween the innermost and next innermost finger contacts;

FIG. 33 is a plot of the inter-hand separation factor versus theestimated distance between the right thumb and left thumb;

FIG. 34 is a flow chart of hand motion component extraction;

FIG. 35 is a diagram of typical finger trajectories when hand iscontracting;

FIG. 36 is a flow chart of radial and angular hand velocity extraction;

FIG. 37 is a flow chart showing extraction of translational handvelocity components;

FIG. 38 is a flow chart of differential hand pressure extraction;

FIG. 39A is a flow chart of the finger synchronization detection loop;

FIG. 39B is a flow chart of chord tap detection;

FIG. 40A is a flow chart of the chord motion recognition loop;

FIG. 40B is a flow chart of chord motion event generation;

FIG. 41 is a flow chart of key layout morphing;

FIG. 42 is a flow chart of the keypress detection loop;

FIG. 43A is a flow chart of the keypress acceptance and transmissionloop; and

FIG. 43B is a flow chart of typematic emulation.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the present preferredembodiments of the invention, examples of which are illustrated in theaccompanying drawings. Wherever possible the same reference numbers willbe used throughout the drawings to refer to the same or like parts.

FIG. 1 is a system block diagram of the entire, integrated manual inputapparatus. Sensor embedded in the multi-touch surface 2 detect proximityof entire flattened hands 4, fingertips thumbs, palms, and otherconductive touch devices to the surface 2. In a preferred embodiment,the surface is large enough to comfortably accommodate both hands 4 andis arched to reduce forearm pronation.

In alternative embodiments the multi-touch surface 2 may be large enoughto accommodate motion of one hand, but may be flexible so it can befitted to an armrest or clothing.

Electronic scanning hardware 6 controls and reads from each proximitysensor of a sensor array. A calibration module 8 constructs a rawproximity image from a complete scan of the sensor array and subtractsoff any background sensor offsets. The background sensor offsets cansimply be a proximity image taken when nothing is touching the surface.

The offset-corrected proximity image is then passed on to the contacttracking and identification module 10, which segments the image intodistinguishable hand-surface contacts, tracks and identifies them asthey move through successive images.

The paths of identified contacts are passed on to a typing recognizermodule 12, finger synchronization detection module 14, motion componentextraction module 16, and pen grip detection module 17, which containsoftware algorithms to distinguish hand configurations and respond todetected hand motions.

The typing recognizer module 12 responds to quick presses and releasesof fingers which are largely asynchronous with respect to the activityof other fingers on the same hand. It attempts to find the key regionnearest to the location of each finger tap and forwards the key symbolsor commands associated with the nearest key region to the communicationinterface module 20.

The finger synchronization detector 14 checks the finger activity withina hand for simultaneous presses or releases of a subset of fingers. Whensuch simultaneous activity is detected it signals the typing recognizerto ignore or cancel keystroke processing for fingers contained in thesynchronous subset. It also passes on the combination of fingeridentities in the synchronous subset to the chord motion recognizer 18.

The motion component extraction module 16 computes multiple degrees offreedom of control from individual finger motions during easilyperformable hand manipulations on the surface 2, such as handtranslations, hand rotation about the wrist, hand scaling by graspingwith the fingers, and differential hand tilting.

The chord motion recognizer produces chord tap or motion eventsdependent upon both the synchronized finger subset identified by thesynchronization detector 14 and on the direction and speed of motionextracted in 16. These events are then posted to the host communicationinterface 20.

The pen grip detection module 17 checks for specific arrangements ofidentified hand contacts which indicate the hand is configured as ifgripping a pen. If such an arrangement is, detected, it forwards themovements of the gripping fingers as inking events to the hostcommunication interface 20. These inking events can either lay digitalink on the host computer display for drawing or signature capturepurposes, or they can be further interpreted by handwriting recognitionsoftware which is well known in the art. The detailed steps within eachof the above modules will be further described later.

The host communication interface keeps events from both the typingrecognizer 12 and chord motion recognizer 18 in a single temporallyordered queue and dispatches them to the host computer system 22. Themethod of communication between the interface 20 and host computersystem 22 can vary widely depending on the function and processing powerof the host computer. In a preferred embodiment, the communication wouldtake place over computer cables via industry standard protocols such asApple Desktop Bus, PS/2 keyboard and mouse protocol for PCs, orUniversal Serial Bus (USB). In alternative embodiments the softwareprocessing of modules 10-18 would be performed within the host computer22. The multi-touch surface apparatus would only contain enough hardwareto scan the proximity sensor array 6, form proximity images 8, andcompress and send them to the host computer over a wireless network. Thehost communication interface 20 would then play the role of devicedriver on the host computer, conveying results of the proximity imagerecognition process as input to other applications residing on the hostcomputer system 22.

In a preferred embodiment the host computer system outputs to a visualdisplay device 24 so that the hands and fingers 4 can manipulategraphical objects on the display screen. However, in alternativeembodiments the host computer might output to an audio display orcontrol a machine such as a robot.

The term “proximity” will only be used in reference to the distance orpressure between a touch device such as a finger and the surface 2, notin reference to the distance between adjacent fingers. “Horizontal” and“vertical” refer to x and y directional axes within the surface plane.Proximity measurements are then interpreted as pressure in a z axisnormal to the surface. The direction “inner” means toward the thumb of agiven hand, and the direction “outer” means towards the pinky finger ofa given hand. For the purposes of this description, the thumb isconsidered a finger unless otherwise noted, but it does not count as afingertip. “Contact” is used as a general term for a hand part when ittouches the surface and appears in the current proximity image, and forthe group and path data structures which represent it.

FIG. 2 is a schematic diagram of a device that outputs a voltage 58dependent on the proximity of a touch device 38 to a conductive senseelectrode 33. The proximity sensing device includes two electricalswitching means 30 and 31 connected together in series having a commonnode 48, an input node 46, and an output node 45. A thin dielectricmaterial 32 covers the sensing electrode 33 that is electricallyconnected to the common node 48. A power supply 34 providing anapproximately constant voltage is connected between reference ground andthe input node 46. The two electrical switches 30 and 31 gate the flowof charge from the power supply 34 to an integrating capacitor 37. Thevoltage across the integrating capacitor 37 is translated to anothervoltage 58 by a high-impedance voltage amplifier 35. The plates of theintegrating capacitor 37 can be discharged by closing electrical switch36 until the voltage across the integrating capacitor 37 is near zero.The electrical switches 30 and 31 are opened and closed in sequence butare never closed at the same time, although they may be opened at thesame time as shown in FIG. 2. Electrical switch 30 is referred to as theinput switch; electrical switch 31 is referred to as the output switch;and, electrical switch 36 is referred to as the shorting switch.

The proximity sensing device shown in FIG. 2 is operated by closing andopening the electrical switches 30, 31, and 36 in a particular sequenceafter which the voltage output from the amplifier 58, which is dependenton the proximity of a touch device 38, is recorded. Sensor operationbegins with all switches in the open state as shown in FIG. 2. Theshorting switch 36 is then closed for a sufficiently long time to reducethe charge residing on the integrating capacitor 37 to a low level. Theshorting switch 37 is then opened. The input switch 30 is then closedthus allowing charge to flow between the power supply and the commonnode 48 until the voltage across the input switch 30 becomes zero.Charge Q will accumulate on the sensing electrode 33 according toQ=V(e*A)/D  (1)where V is the voltage of the power supply 34, e is the permittivity ofthe dielectric sensing electrode cover 32 and the air gap between thecover and the touch device 38, D is the thickness of this dielectricregion, and A is the overlap area of the touch device 38 and the sensingelectrode 33. Therefore the amount of charge accumulating on the sensingelectrode 33 will depend, among other things, on the area of overlap ofthe touch device 38 and the sensing electrode 33 and the distancebetween the touch device 38 and the sensing electrode 33. The inputswitch 30 is opened after the voltage across it has become zero, ornearly so. Soon after input switch 30 is opened the output switch 31 isclosed until the voltage across it is nearly zero. Closing the outputswitch 31 allows charge to flow between the sensing electrode 33 and theintegrating capacitor 37 resulting in a voltage change across theintegrating capacitor 37 according to:deltaV=(V−Vc)/(1+C*D/e*A)  (2)where Vc is the voltage across the integrating capacitor 37 before theoutput switch 31 was closed, C is the capacitance of the integratingcapacitor 37, and A and D are equal to their values when input switch 30was closed as shown in Equation 1. Multiple switchings of the input 30and output 31 switches as described above produce a voltage on theintegrating capacitor 37 that reflects the proximity of a touch device38 to the sensing electrode 33.

FIG. 3A is a schematic diagram of the proximity sensor in which theshorting transistor 36 and the voltage-to-voltage translation device 35are replaced by a resistor 40 and a current-to-voltage translationdevice 41, respectively. The integrating function of capacitor 37 shownin FIG. 2 is, in this variation of the proximity sensor, carried out bythe capacitor 39 shown in FIG. 3A. Those skilled in the art will seethat this variation of the proximity sensor produces a more linearoutput 58 from multiple switchings of the input and output switches,depending on the relative value of the resistor 40. Alternatively, theresistor 40 can be replaced by a shorting switch 69 (cf. FIG. 3B) toimprove linearity. Although, the circuits shown in FIG. 3 provide a morelinear output than the circuit shown in FIG. 2 the circuits of FIG. 3generally require dual power supplies while the circuit of FIG. 2requires only one.

The electrical switches shown in, FIG. 2 can be implemented with varioustransistor technologies: discrete, integrated, thin film, thick film,polymer, optical, etc. One such implementation is shown in FIG. 4A wherefield effect transistors (FETs) are used as the input 30, output 31, andshorting 36 switches. The FETs are switched on and off by voltagesapplied to their gate terminals (43, 44, and 55). For the purpose ofthis description we will assume the FET is switched on when its gatevoltage is logic 1 and switched off when its gate voltage is logic 0. Acontroller 42 is used to apply gate voltages as a function of time asshown in FIG. 4B. In this example, a sequence of three pairs of pulses(43 and 44) are applied to the input and output transistor gates. Eachpair of pulses 43 and 44 produces a voltage change across theintegrating capacitor 37 as shown in Equation 2. The number of pulsepairs applied to input 43 and output 44 gates depends on the desiredvoltage across integrating capacitor 37. In typical applications thenumber is between one and several hundred pulse-pairs.

FIG. 5 shows the proximity sensor circuitry appropriate for use in asystem comprising an array of proximity sensors 47 as in a multi-touchsurface system. The proximity sensor 47 consists of the input transistor30, the output transistor 31, the sensing electrode 33, the dielectriccover 32 for the sensing electrode 33, and conductive traces 43, 44, 45,and 46. The conductive traces are arranged so as to allow the proximitysensors 47 comprising a 2D array to be closely packed and to share thesame conductive traces, thus reducing the number of wires needed in asystem. FIG. 6 shows an example of such a system where the input nodes46 of all proximity sensors are connected together and connected to apower supply 34. The output nodes 45 of all proximity sensors areconnected together and connected to a single integrating capacitor 37, asingle shorting transistor 36, and a single voltage-to-voltage amplifier35. In this implementation, a single proximity sensor 47 is enabled at atime by applying a logic 1 signal first to its input gate 43 and then toits output gate 44. This gating of a single proximity sensor 47 one at atime is done by input gate controller 50 and output gate controller 51.For example, to enable the proximity sensor 47 in the lower right cornerthe input gate controller 50 would output a logic one pulse onconductive trace 43 a. This is followed by a logic one pulse onconductive trace 44 h produced by output gate controller 51. Repetitionof this pulse as shown in FIG. 4B would cause charge to build up onintegrating capacitor 37 and a corresponding voltage to appear at theoutput of the amplifier 58. The entire array of proximity sensors 47 isthus scanned by enabling a single sensor at a time and recording itsoutput.

FIG. 7A is a schematic of typical circuitry useful for converting theproximity sensor output 58 to a digital code appropriate for processingby computer. The proximity sensor output 58 is typically non-zero evenwhen there is no touch device (e.g., ref. no. 38 in FIG. 2) nearby. Thisnon-zero signal is due to parasitic or stray capacitance present at thecommon node 48 of the proximity sensor and is of relatively constantvalue. It is desirable to remove this non-zero background signal beforeconverting the sensor output 58 to a digital code. This is done by usinga differential amplifier 64 to subtract a stored record of thebackground signal 68 from the sensor output 58. The resulting differencesignal 65 is then converted to a digital code by an ADC (analog todigital converter) 60 producing a K-bit code 66. The stored backgroundsignal is first recorded by sampling the array of proximity sensors 47(FIG. 6) with no touch devices nearby and storing a digital codespecific for each proximity sensor 47 in a memory device 63. Theparticular code corresponding to the background signal of each proximitysensor is selected by an M-bit address input 70 to the memory device 63and applied 69 to a DAC (digital to analog converter) 61.

The 2D array of proximity sensors 47 shown in FIG. 6 can be connected ingroups so as to improve the rate at which the entire array is scanned.This is illustrated in FIG. 8 where the groups are arranged as columnsof proximity sensors. In this approach, the input nodes of the proximitysensors are connected together and connected to a power supply 34, as inFIG. 6. The output gates 44 are also connected in the same way. However,the input gates 43 are now all connected together and the output nodes45 are connected to only those proximity sensors 47 within a row and toa dedicated voltage amplifier 35. With this connection method, all ofthe proximity sensors in a column are enabled at a time, thus reducingthe time to scan the array by a factor N, where N is the number ofproximity sensors in a group. The outputs 58 a-h could connect todedicated converter circuitry as shown in FIG. 7A or alternatively eachoutput 58 a-h could be converted one at a time using the circuitry shownin FIG. 7B. In this figure, the output signals from each group 58 a-hare selected one at a time by multiplexer 62 and applied to the positiveinput of the differential amplifier 64. With this later approach, it isassumed that the ADC 60 conversion time is much faster than the sensorenable time, thus providing the suggested speed up in sensor arrayscanning.

FIG. 9 shows a typical circuit useful for the control of the proximitysensor's output gate 44. It consists of three input signals 75, 76, 78and two output signals 44, 77. The output gate signal 44 is logic 1 whenboth inputs to AND gate 79 are logic 1. The AND input signal 77 becomeslogic 1 if input signal 76 is logic 1 when input signal 78 transitionsfrom logic 0 to logic 1, otherwise it remains logic 0. A linear array ofthese circuits 81 can be connected end-to-end to enable the output gatesof a single group of proximity sensors at a time as shown in FIG. 8.

FIG. 10 shows a cover for the multi-touch surface 89 that permits thesystem to be sensitive to pressure exerted by non-conducting touchobjects (e.g., gloved fingers) contacting the multi-touch surface. Thiscover comprises a deformable dielectric touch layer 85, a deformableconducting layer 86, and a compliant dielectric layer 87. The touchsurface 85 would have a symbol set printed on it appropriate for aspecific application, and this surface could be removed and replacedwith another one having a different symbol set. The conducting layer 86is electrically connected 88 to the reference ground of the proximitysensor's power supply 34. When a touch object presses on the top surface85 it causes the conducting surface 86 under the touch device to movecloser to the sensing electrode 33 of the proximity sensor. This resultsin a change in the amount of charge stored on the sensing electrode 33and thus the presence of the touch object can be detected. The amount ofcharge stored will depend on the pressure exerted by the touch object.More pressure results in more charge stored as indicated in Equation 1.

To obtain a softer touch surface on the multi-touch device a thicker andmore, compliant dielectric cover could be used. However, as thedielectric thickness increases the effect of the touch device on thesensing electrodes 33 spreads out thus lowering spatial resolution. Acompliant anisotropically-conducting material can be used to counterthis negative effect while also providing a soft touch surface. FIG. 11shows a cover in which a compliant anisotropically-conducting material90 is set between a thin dielectric cover 85 and the sensing electrodes33. If the conductivity of the compliant material 90 is oriented mostlyin the vertical direction, the image formed by a touch device on thesurface 85 will be translated without significant spreading to thesensing electrodes 33, thus preserving spatial resolution whileproviding a compliant touch surface.

FIG. 12 shows a cross section of a multi-touch surface that senses boththe proximity and pressure of a touch device. The touch layer 85 is athin dielectric that separates touch devices from the sensing electrodes33. Proximity sensing is relative to this surface. The electrodes 33 andassociated switches and conductors are fabricated on a compliantmaterial 89 which is attached to a rigid metal base 92. The metal base92 is electrically connected 88 to the reference ground of the proximitysensor's power supply 34. When a touch device presses on the touchsurface 85 it causes the sensing electrodes 33 directly below to movecloser to the rigid metal base 92. The distance moved depends on thepressure applied and thus the pressure exerted by a touch device can bedetected as described before.

To illustrate typical properties of hand contacts as they appear inproximity images, FIGS. 13-15 contain sample images captured by aprototype array of parallelogram-shaped electrodes. Shading of eachelectrode darkens to indicate heightened proximity signals as flesh getscloser to the surface, compresses against the surface due to handpressure, and overlaps the parallelogram more completely. Note that theresolution of these images is in no way intended to limit the scope ofthe invention, since certain applications such as handwritingrecognition will clearly require finer electrode arrays than indicatedby the electrode size in these sample images. In the discussion thatfollows, the proximity data measured at one electrode during aparticular scan cycle constitutes one “pixel” of the proximity imagecaptured in that scan cycle.

FIG. 13 shows a right hand flattened against the surface with fingersoutstretched. At the far left is the oblong thumb 201 which tends topoint off at about 120-degrees. The columnar blobs arranged in an arcacross the top of the image are the index finger 202, middle finger 203,ring finger 204 and pinky finger 205. Flesh from the proximal fingerjoint, or proximal phalanges 209, will appear below each fingertip ifthe fingers are fully extended. The inner 207 and outer 206 palm heelscause the pair of very large contacts across the bottom of the image.Forepalm calluses 213 are visible at the center of the hand if the palmis fully flattened. This image shows that all the hand contacts areroughly oval-shaped, but they differ in pressure, size, orientation,eccentricity and spacing relative to one another. This image includesall of the hand parts which can touch the surface from the bottom of onehand but in many instances only a few of these parts will be touchingthe surface, and the fingertips may roam widely in relation to the palmsas fingers are flexed and extended.

FIG. 14 shows another extreme in which the hand is partially closed. Thethumb 201 is adducted toward the fingertips 202-208 and the fingers areflexed so the fingertips come down normal instead of tangential to thesurface. The height and intensity of fingertip contacts is lessenedsomewhat because the boney tip rather than fleshy pulp pad is actuallytouching the surface, but fingertip width remains the same. Adjacentfingertips 202-205 and thumb 201 are so close together as to bedistinguishable only by slight proximity valleys 210 between them. Theproximal phalange finger joints are suspended well above the surface anddo not appear in the image, nor do the forepalm calluses. The palm heels206, 207 are somewhat shorter since only the rear of the palm can touchthe surface when fingers are flexed, but the separation between them isunchanged. Notice that the proximity images are uncluttered bybackground objects. Unlike optical images, only conductive objectswithin a few millimeters of the surface show up at all.

FIG. 15 is a proximity image of a right hand in a pen gripconfiguration. The thumb 201 and index fingertip 202 are pinchedtogether as if they were holding a pen but in this case they aretouching the surface instead. Actually the thumb and index finger appearthe same here as in FIG. 14. However, the middle 203, ring 204, andpinky 205 fingers are curled under as if making a fist, so the knucklesfrom the top of the fingers actually touch the surface instead of thefinger tips. The curling under of the knuckles actually places thembehind the pinched thumb 201 and index fingertip 202 very close to thepalm heels 206, 207. The knuckles also appear larger than the curledfingertips of FIG. 14 but the same size as the flattened fingertips inFIG. 13. These differences in size and arrangement will be measured bythe pen grip detector 17 to distinguish this pen grip configuration fromthe closed and flattened hand configurations.

FIG. 16 represents the data flow within the contact tracking andidentification module 10. The image segmentation process 241 takes themost recently scanned proximity image data 240 and segments it intogroups of electrodes 242 corresponding to the distinguishable hand partsof FIG. 13. The filtering and segmentation rules applied in particularregions of the image are partially determined by feedback of theestimated hand offset data 252. The image segmentation process 241outputs a set of electrode group data structures 242 which areparameterized by fitting an ellipse to the positions and proximitymeasurements of the electrodes within each group.

The path tracking process 245 matches up the parameterized electrodegroups 242 with the predicted continuations of contact path datastructures 243 extracted from previous images. Such path trackingensures continuity of contact representation across proximity images.This makes it possible to measure the velocity of individual handcontacts and determine when a hand part lifts off the surface,disappearing from future images. The path tracking process 245 updatesthe path positions, velocities, and contact geometry features from theparameters of the current groups 242 and passes them on to the contactidentification processes 247 and 248. For notational purposes, groupsand unidentified paths will be referred to by data structure names ofthe form Gi and Pi respectively, where the indices i are arbitraryexcept for the null group G0 and null path P0. Particular group and pathparameters will be denoted by subscripts to these structure names andimage scan cycles will be denoted by bracketed indices, so that, forexample, P2 _(s)[n] represents the horizontal position of path 2 in thecurrent proximity image, and P2 _(x)[n−1] represents the position in theprevious proximity image. The contact identification system ishierarchically split into a hand identification process 247 andwithin-hand finger and palm identification process 248. Given a handidentification for each contact, the finger and palm identificationprocess 248 utilizes combinatorial optimization and fuzzy patternrecognition techniques to identify the part of the hand causing eachsurface contact. Feedback of the estimated hand offset helps identifyhand contacts when so few contacts appear in the image that the overallhand structure is not apparent.

The hand identification process 247 utilizes a separate combinatorialoptimization algorithm to find the assignment of left or right handidentity to surface contacts which results in the most biomechanicallyconsistent within-hand identifications. It also receives feedback of theestimated hand and finger offsets 252, primarily for the purpose oftemporarily storing the last measured hand position after fingers in ahand lift off the surface. Then if the fingers soon touch back down inthe same region they will more likely receive their previous handidentifications.

The output of the identification processes 247 and 248 is the set ofcontact paths with non-zero hand and finger indices attached. Fornotational purposes identified paths will be referred to as F0 for theunidentified or null finger, F1 for the thumb 201, F2 for the indexfinger 202, F3 for the middle finger 203, F4 for the ring finger 204, F5for the pinky finger 205, F6 for the outer palm heel 206. F7 for theinner palm heel 207, and F8 for the forepalm calluses 208. To denote aparticular hand identity this notation can be prefixed with an L forleft hand or R for right hand, so that, for example, RF2 denotes theright index finger path. When referring to a particular hand as a whole.LH denotes the left hand and RH denotes the right hand. In the actualalgorithms left hand identity is represented by a −1 and right hand by+1, so it is easy to reverse the handedness of measurements taken acrossthe vertical axis of symmetry.

It is also convenient to maintain for each hand a set of bitfield dataregisters for which each bit represents touchdown, continued contact orliftoff of a particular finger. Bit positions within each bit fieldcorrespond to the hand part indices above. Such registers can quickly betested with a bit mask to determine whether a particular subset offingers has touched down. Alternatively, they can be fed into a lookuptable to find the input events associated with a particular finger chord(combination of fingers). Such finger identity bitfields are neededprimarily by the synchronization detector 14 and chord motion recognizer18.

The last process within the tracking and identification subsystem is thehand position estimator 251, which as described above provides biasingfeedback to the identification and segmentation processes. The handposition estimator is intended to provide a conservative guess 252 oflateral hand position under all conditions including when the hand isfloating above the surface without touching. In this case the estimaterepresents a best guess of where the hand will touch down again. Whenparts of a hand are touching the surface, the estimate combines thecurrent position measurements of currently identified hand parts withpast estimates which may have been made from more or less reliableidentifications.

The simplest but inferior method of obtaining a hand positionmeasurement would be to average the positions of all the hand's contactsregardless of identity. If hand parts 201-207 were all touching thesurface as in FIG. 13 the resulting centroid would be a decent estimate,lying somewhere under the center of the palm since the fingers and palmheels typically form a ring around the center of the palm. However,consider when only one hand contact is available for the average. Theestimate would assume the hand center is at the position of this lonecontact, but if the contact is from the right thumb the hand centerwould actually be 4-8 cm to the right, or if the contact is from a palmheel the hand center is actually 4-6 cm higher, or if the lone contactis from the middle finger the hand center should actually be actually4-6 cm lower.

FIG. 17 shows the detailed steps within the hand position estimator 251.The steps must be repeated for each hand separately. In a preferredembodiment, the process utilizes the within-hand contact identifications(250) to compute (step 254) for each contact an offset between themeasured contact position (Fi_(x)[n],Fi_(y)[n]) and the default positionof the particular finger or palm heel (Fi_(defx),Fi_(defy)) with handpart identity i. The default positions preferably correspond to fingerand palm positions when the hand is in a neutral posture with fingerspartially closed, as when resting on home row of a keyboard. Step 255averages the individual contact offsets to obtain a measured handoffset, (H_(mox)[n],H_(moy)[n]):

$\begin{matrix}{{H_{mox}\lbrack n\rbrack} = \frac{\sum\limits_{i = 1}^{i = 7}{{{Fi}_{mow}\lbrack n\rbrack}\left( {{{Fi}_{x}\lbrack n\rbrack} - {Fi}_{defx}} \right)}}{\sum\limits_{i = 1}^{i = 7}{{Fi}_{mow}\lbrack n\rbrack}}} & (3) \\{{H_{moy}\lbrack n\rbrack} = \frac{\sum\limits_{i = 1}^{i = 7}{{{Fi}_{mow}\lbrack n\rbrack}\left( {{{Fi}_{y}\lbrack n\rbrack} - {Fi}_{defy}} \right)}}{\sum\limits_{i = 1}^{i = 7}{{Fi}_{mow}\lbrack n\rbrack}}} & (4)\end{matrix}$

Preferably the weighting Fi_(mow)[n] of each finger and palm heel isapproximately its measured total proximity, i.e., Fi_(mow)[n]=Fi [n].This ensures that lifted fingers, whose proximity is zero, have noinfluence on the average, and that contacts with lower than normalproximity, whose measured positions and identities are less accurate,have low influence. Furthermore, if palm heels are touching, their largetotal proximities will dominate the average. This is beneficial becausethe palm heels, being immobile relative to the hand center compared tothe highly flexible fingers, supply a more reliable indication ofoverall hand position. When a hand is not touching the surface, i.e.,when all proximities are zero, the measured offsets are set to zero.This will cause the filtered hand position estimate below to decaytoward the default hand position.

As long as the contact identifications are correct, this hand positionmeasurement method eliminates the large errors caused by assuming lonecontacts originate from the center of the hand. Flexing of fingers fromtheir default positions will not perturb the measured centroid more thana couple centimeters. However, this scheme is susceptible to contactmisidentification, which can cause centroid measurement errors of up to8 cm if only one hand part is touching. Therefore, the current measuredoffsets are not used directly, but are averaged with previous offsetestimates (H_(eox)[n−1],H_(eoy)[n−1]) using a simple first-orderautoregressive filter, forming current offset estimates(H_(eox)[n],H_(eoy)[n]).

Step 256 adjusts the filter pole H_(oa)[n] according to confidence inthe current contact identifications. Since finger identificationsaccumulate reliability as more parts of the hand contact the surface onesimple measure of identification confidence: is the number of fingerswhich have touched down from the hand since the hand last left thesurface. Contacts with large total proximities also improveidentification reliability because they have strong disambiguatingfeatures such as size and orientation. Therefore H_(oa)[n] is setroughly proportional to the maximum finger count plus the sum of contactproximities for the hand. H_(oa)[n] must of course be normalized to bebetween zero and one or the filter will be unstable. Thus whenconfidence in contact identifications is high, i.e., when many parts ofthe hand firmly touch the surface, the autoregressive filter favors thecurrent offset measurements. However, when only one or two contacts havereappeared since hand liftoff, the filter emphasizes previous offsetestimates in the hope that they were based upon more reliableidentifications.

The filtered offsets must also maintain a conservative estimate of handposition while the hand is floating above the surface for optimalsegmentation and identification as the hand touches back down. If a handlifts off the surface in the middle of a complex sequence of operationsand must, quickly touch down again, it will probably touch down close towhere it lifted off. However, if the operation sequence has ended, thehand is likely to eventually return to the neutral posture, or defaultposition, to rest. Therefore, while a hand is not touching the surface,H_(oa)[n] is made small enough that the estimated offsets graduallydecay to zero at about the same rate as a hand lazily returns to defaultposition.

When H_(oa)[n] is made small due to low identification confidence, thefilter tracking delay becomes large enough to lag behind a pair ofquickly moving fingers by several centimeters. The purpose of the filteris to react slowly to questionable changes in contact identity, not tosmooth contact motion. This motion tracking delay can be safelyeliminated by adding the contact motion measured between images to theold offset estimate. Step 257 obtains motion from the average,(H_(nvx)[n],H_(nvy)[n]) of the current contact velocities:

$\begin{matrix}{{H_{mvx}\lbrack n\rbrack} = \frac{\sum\limits_{i = 1}^{i = 7}{{{Fi}_{mow}\lbrack n\rbrack}{{Fi}_{vx}\lbrack n\rbrack}}}{\sum\limits_{i = 1}^{i = 7}{{Fi}_{mow}\lbrack n\rbrack}}} & (5) \\{{H_{mvy}\lbrack n\rbrack} = \frac{\sum\limits_{i = 1}^{i = 7}{{{FI}_{mow}\lbrack n\rbrack}{{Fi}_{vy}\lbrack n\rbrack}}}{\sum\limits_{i = 1}^{i = 7}{{Fi}_{mow}\lbrack n\rbrack}}} & (6)\end{matrix}$

The current contact velocities. (Fi_(vx)[n],F_(vy)[n]), are retrievedfrom the path tracking process 245, which measures them independent offinger identity. Step 258 updates the estimated hand offsets(H_(eox)[n],H_(eoy)[n]) using the complete filter equations:H _(eox) [n]=H _(oa) [n]H _(mox) [n]+(1−H _(oa) [n])(H _(eox) [n−1]+H_(mox) [n]Δt)  (7)H _(eoy) [n]=H _(oa) [n]H _(moy) [n]+(1−H _(oa) [n])(H _(eoy) [n−1]+H_(moy) [n]Δt)  (8)

Finally, to provide a similarly conservative estimate of the positionsof particular fingers step 259 computes individual finger offsets(Fi_(eox)[n],Fi_(eoy)[n]) from the distance between identified contactsand their corresponding default finger positions less the estimated handoffsets. For each identifiable contact i, the offsets are computed as:Fi _(eox) [n]=H _(oa) [n](H _(mox) [n]+Fi _(x) [n]−Fi _(defx))+(1−H_(oa) [n])(Fi _(eox) [n−1]+Fi _(vx) [n]Δt)  (9)FI _(eoy) [n]=H _(oa) [n](H _(moy) [n]+Fi _(y) [n]−Fi _(defy))+(1−H_(oa) [n])(Fi _(eoy) [n−1]+Fi _(vy) [n]Δt)  (10)

These finger offsets reflect deviations of finger flexion and extensionfrom the neutral posture. If the user places the fingers in an extremeconfiguration such as the flattened hand configuration, the collectivemagnitudes of these finger offsets can be used as an indication of userhand size and finger length compared to the average adult.

The parameters (H_(eox)[n],H_(eoy)[n]) and (Fi_(eox)[n],Fi_(eoy)[n]) foreach hand and finger constitute the estimated hand and finger offsetdata 252, which is fed back to the segmentation and identificationprocesses during analysis of the next proximity image. If the otherprocesses need the estimate in absolute coordinates, they can simply add(step 260) the supplied offsets to the default finger positions, but inmany cases the relative offset representation is actually moreconvenient.

It should be clear to those skilled in the art that many improvementscan be made to the above hand position estimation procedure which remainwell within the scope of this invention, especially in the manner ofguessing the position of lifted hands. One improvement is to make theestimated hand offsets decay toward zero at a constant speed when a handis lifted rather than decay exponentially. Also, the offset computationsfor each hand have been independent as described so far. It is actuallyadvantageous to impose a minimum horizontal separation between theestimated left hand position and estimated right hand position such thatwhen a hand such as the right hand slides to the opposite side of theboard while the other hand is lifted, the estimated position of theother hand is displaced. In this case the estimated position of thelifted left hand would be forced from default to the far left of thesurface, possibly off the surface completely. If the right hand islifted and the left is not, an equation like the following can beapplied to force the estimated right hand position out of the way:Rh _(eox) [n]:=min(RH _(eox) [n],(LF1_(defx) −RF1_(defx))+Lh _(eox)[n]+min_(—) hard _(—) sep)  (11)where (LF1_(defx)−RF1_(defx)) is the default separation between left andright thumbs, is the minimum horizontal separation to be imposed, andLH_(eox)[n] is the current estimated offset of the left hand.

FIG. 18 represents the data flow within the proximity image segmentationprocess 241. Step 262 makes a spatially smoothed copy 263 of the currentproximity image 240 by passing a two-dimensional diffusion operator orGaussian kernel over it. Step 264 searches the smoothed image 263 forlocal maximum pixels 265 whose filtered proximity exceeds a significancethreshold and exceeds the filtered proximities of nearest neighborpixels. The smoothing reduces the chance that an isolated noise spike ona single electrode will result in a local maximum—which exceeds thesignificance threshold, and consolidates local maxima to about one perdistinguishable fleshy contact.

Process 268 then constructs a group of electrodes or pixels whichregister significant proximity around each local maximum pixel bysearching outward from each local maximum for contact edges. Eachelectrode encountered before reaching a contact boundary is added to thelocal maximum's group. FIG. 19 shows the basic boundary electrode searchpattern for an example contact boundary 274. In this diagram, anelectrode or image pixel lies at the tip of each arrow. The searchstarts at the local maximum pixel 276, proceeds to the left pixels 277until the boundary 274 is detected. The last pixel before the boundary278 is marked as an edge pixel, and the search resumes to the right 279of the local maximum pixel 276. Once the left and right edges of thelocal maximum's row have been found, the search recurses to the rowsabove and below, always starting 281 in the column of the pixel in theprevious row which had the greatest proximity. As the exampleillustrates, the resulting set of pixels or electrodes is connected inthe mathematical sense but need not be rectangular. This allows groupsto closely fit the typical oval-shape of flesh contacts without leavingelectrodes out or including those from adjacent contacts.

If contacts were small and always well separated, edges could simply beestablished wherever proximity readings fell to the background level.But sometimes fingertips are only separated by a slight valley orshallow saddle point 210. To segment adjacent fingertips the partialminima of these valleys must be detected and used as group boundaries.Large palm heel contacts, on the other hand, may exhibit partial minimadue to minor nonuniformities in flesh proximity across the contact. Ifall electrodes under the contact are to be collected in a single group,such partial minima must be ignored. Given a hand position estimate thesegmentation system can apply strict edge detection rules in regions ofthe image where fingertips and thumb are expected to appear but applysloppy edge detection rules in regions of the image where palms areexpected to, appear. This ensures that adjacent fingertips are notjoined into a single group and that each palm heel is not broken intomultiple groups.

Step 266 of FIG. 18 defines the positions of these segmentation regionsusing the hand position estimates 252 derived from analyses of previousimages. FIG. 20A shows the extent of the strict and sloppy segmentationregions while the hands are in their default positions, making estimatedoffsets for both hands zero. Plus signs in the diagram 252 indicate theestimated position of each finger and palm heel in each hand.Rectangular outlines in the lower corners represent the left 284 andright 286 sloppy segmentation regions where partial minima are largelyignored. The T-shaped region remaining is the strict segmentation region282, where proximity saddle points must serve as contact boundaries. Asa preferred embodiment the sloppy regions are rectangular, their innerboundaries 285 are placed just inside of the columns where the indexfingers 202 are expected to lie, and the upper boundaries 287 are placedat the estimated vertical levels of their respective thumbs 201. Theouter and lower boundaries of the sloppy regions are determined by theoutside edges of the surface. Due to the decay in estimated hand offsetsafter hands leave the surface, the sloppy segmentation regions return tothe positions shown after the hands have stayed off the surface a fewseconds, regardless of hand position at liftoff. FIG. 20B shows how thesloppy regions follow the estimated hand positions 252 as the right handmoves toward the upper left and the left hand moves toward the lowerleft. This ensures that the palms and only the palms fall in the sloppyregions as long as the hand position estimates are correct.

FIG. 20C shows that the left sloppy region 284 is moved left off thesurface entirely when the left hand is lifted off the surface and theright hand slides to the left side of the surface. This prevents thefingers of one hand from entering the sloppy segmentation region of theopposite hand. This effect is implemented by imposing a minimumhorizontal separation between the sloppy regions and, should the regionsget too close to one another, letting the hand with the most surfacecontacts override the estimated position of the hand with fewercontacts. FIG. 21 is a detailed flow chart of the edge tests which areapplied at each searched electrode depending on whether the electrode isin a strict or sloppy segmentation region. Decision diamond 290 checkswhether the unsmoothed proximity of the electrode is greater than thebackground proximity levels. If not, the electrode is labeled an edgeelectrode in step 304 regardless of the segmentation region or searchdirection, and in step 305 the search returns to the row maximum torecurse in another direction. If the unsmoothed proximity is significantfarther tests are applied to the smoothed proximity of neighboringelectrodes depending on whether decision diamond 292 decides the searchelectrode is in a sloppy or strict region.

If a strict region search is advancing horizontally within a row,decision diamond 306 passes to decision diamond 308 which tests whetherthe electrode lies in a horizontal or diagonal partial minimum withrespect to its nearest neighbor electrodes. If so, a proximity valleybetween adjacent fingers has probably been detected, the electrode islabeled as an edge 314 and search resumes in other directions 305. Ifnot, the search continues on the next electrode in the row 302. If astrict region search is advancing vertically to the next row, decisiondiamond 306 passes to decision diamond 310 which tests whether theelectrode lies in a vertical partial minimum with respect to thesmoothed proximity of its nearest neighbor electrodes. If so, aproximity valley between a finger and the thumb has probably beendetected, the electrode is labeled as an edge 312 and search resumes inother directions 305. If not, the search continues into the next row302. If decision diamond 294 determines that a sloppy region search isadvancing horizontally within a row, stringent horizontal minimum testsare performed to check for the crease or proximity valley between theinner and outer palm heels. To qualify, the electrode must be more thanabout 2 cm horizontal distance from the originating local maximum, aschecked by decision diamond 296. Also the electrode must be part of atall valley or partial horizontal minimum which extends to the rowsabove and below and the next-nearest neighbors within the row, aschecked by decision diamond 298. If so, the electrode is labeled as anedge 300 and search recurses in other directions 305. All other partialminima within the sloppy regions are ignored, so the search continues302 until a background level edge is reached on an upcoming electrode.

In sloppy segmentation regions it is possible for groups to overlapsignificantly because partial minima between local maxima do not act asboundaries. Typically when this happens the overlapping groups are partof a large fleshy contact such as a palm which, even after smoothing,has multiple local maxima. Two groups are defined to be overlapping ifthe search originating local maximum electrode of one group is also anelement of the other group. In the interest of presenting only one groupper distinguishable fleshy contact to the rest of the system, step 270of FIG. 18 combines overlapping groups into single supergroups beforeparameter extraction. Those skilled in the art will realize thatfeedback from high-level analysis of previous images can be applied invarious alternative ways to improve the segmentation process and stilllie well within the scope of this invention. For example, additionalimage smoothing in sloppy segmentation regions could consolidate eachpalm heel contact into a single local maximum which would pass strictsegmentation region boundary tests. Care must be taken with thisapproach however, because too much smoothing can cause finger pairswhich unexpectedly enter sloppy palm regions to be joined into onegroup. Once a finger pair is joined the finger identification process248 has no way to tell that the fingertips are actually not a singlepalm heel, so the finger identification process will be unable tocorrect the hand position estimate or adjust the sloppy regions forproper segmentation of future images.

More detailed forms of feedback than the hand position estimate can beutilized as well. For example, the proximal phalanges (209 in FIG. 13)are actually part of the finger but tend to be segmented into separategroups than the fingertips by the vertical minimum test 310. Thevertical minimum test is necessary to separate the thumb group fromindex fingertip group in the partially closed FIG. 14 and pen grip FIG.15 hand configurations. However, the proximal phalanges of flattenedfingers can be distinguished from a thumb behind a curled fingertip bythe fact that it is very difficult to flatten one long finger withoutflattening the other long fingers. To take, advantage of thisconstraint, a flattened finger flag 267 is set whenever two or more ofthe contacts identified as index through pinky in previous images arelarger than normal, reliably indicating that fingertips are flattening.Then decision diamond 310 is modified during processing of the currentimage to ignore the first vertical minimum encountered during search ofrows below the originating local minimum 276. This allows the proximalphalanges to be included in the fingertip group but prevents fingertipgroups from merging with thumbs or forepalms. The last step 272 of thesegmentation process is to extract shape, size, and position parametersfrom each electrode group. Group position reflects hand contact positionand is necessary to determine finger velocity. The total groupproximity, eccentricity, and orientation are used by higher levelmodules to help distinguish finger, palm, and thumb contacts.

Provided G_(E) is the set of electrodes in group G, e_(z) is theunsmoothed proximity of an electrode or pixel e, and e_(x) and e_(y) arethe coordinates on the surface of the electrode center in centimeters,to give a basic indicator of group position, the proximity-weightedcenter, or centroid, is computed from positions and proximities of thegroup's electrodes:

$\begin{matrix}{G_{z} = {\sum\limits_{e \in G_{E}}e_{z}}} & (12) \\{G_{x} = {\sum\limits_{e \in G_{E}}\frac{e_{z}e_{x}}{G_{z}}}} & (13) \\{G_{y} = {\sum\limits_{e \in G_{E}}\frac{e_{z}e_{y}}{G_{z}}}} & (14)\end{matrix}$

Note that since the total group proximity G_(z) integrates proximityover each pixel in the group, it depends upon both of the size of a handpart, since large hand parts tend to cause groups with more pixels, andof the proximity to or pressure on the surface of a hand part.

Since most groups are convex, their shape is well approximated byellipse parameters. The ellipse fitting procedure requires a unitarytransformation of the group covariance matrix G_(eov) of second momentsQ_(xx), Q_(xy), G_(yy):

$\begin{matrix}{G_{cov} = \begin{bmatrix}G_{xx} & G_{xy} \\G_{yx} & G_{yy}\end{bmatrix}} & (15) \\{G_{xx} = {\sum\limits_{e \in G_{E}}{e_{z}\left( {G_{x} - e_{x}} \right)}^{2}}} & (16) \\{G_{yx} = {G_{xy} = {\sum\limits_{e \in G_{E}}{{e_{z}\left( {G_{x} - e_{x}} \right)}\left( {G_{y} - e_{y}} \right)}}}} & (17) \\{G_{yy} = {\sum\limits_{e \in G_{E}}{e_{x}\left( {G_{y} - e_{y}} \right)}^{2}}} & (18)\end{matrix}$The eigenvalues λ₀ and λ₁ of the covariance matrix G_(eov) determine theellipse axis lengths and orientation G_(θ):

$\begin{matrix}{G_{major} = \sqrt{\lambda_{0}}} & (19) \\{G_{minor} = \sqrt{\lambda_{1}}} & (20) \\{G_{\theta} = {\arctan\left( \frac{\lambda_{0} - G_{xx}}{G_{xy}} \right)}} & (21)\end{matrix}$where G_(θ) is uniquely wrapped into the range (0,180°).

For convenience while distinguishing fingertips from palms at highersystem levels, the major and minor axis lengths are converted via theirratio into an eccentricity G_(∈):

$\begin{matrix}{G_{ɛ} = \frac{G_{major}}{G_{minor}}} & (22)\end{matrix}$

Note that since the major axis length is always greater than or equal tothe minor axis length, the eccentricity will always be greater than orequal to one. Finally, the total group proximity is empiricallyrenormalized so that the typical curled fingertip will have a totalproximity around one:

$\begin{matrix}{{G_{z}\text{:}} = \frac{G_{z}}{Z_{averageFingertip}}} & (23)\end{matrix}$

On low resolution electrode arrays, the total group proximity G_(z) is amore reliable indicator of contact size as well as finger pressure thanthe fitted ellipse parameters. Therefore, if proximity images have lowresolution, the orientation and eccentricity of small contacts are setto default values rather than their measured values, and total groupproximity G_(z) is used as the primary measure of contact size insteadof major and minor axis lengths.

FIG. 22 shows the steps of the path tracking process, which chainstogether those groups from successive proximity images which correspondto the same physical hand contact. To determine where each hand part hasmoved since the last proximity image, the tracking process must decidewhich current groups should be matched with which existing contactpaths. As a general rule, a group and path arising from the same contactwill be closer to one another than to other groups and paths. Also,biomechanical constraints on lateral finger velocity and accelerationlimit how far a finger can travel between images. Therefore a group andpath should not be matched unless they are within a distance known asthe tracking radius of one another. Since the typical lateral separationbetween fingers is greater than the tracking radius for reasonable imagescan rates touchdown and liftoff are easily detected by the fact thattouchdown usually causes a new group to appear outside the trackingradii of existing paths, and liftoff will leave an active path without agroup within its tracking radius. To prevent improper breaking of pathsat high finger speeds each path's tracking radius P_(rtruck) can be madedependent on its existing speed and proximity.

The first step 320 predicts the current locations of surface contactsalong existing trajectories using path positions and velocities measuredfrom previous images. Applying previous velocity to the locationprediction improves the prediction except when a finger suddenly startsor stops or changes direction. Since such high acceleration events occurless often than zero acceleration events, the benefits of velocity-basedprediction outweigh the potentially bad predictions during fingeracceleration. Letting P_(x)[n−1],P_(y)[n−1] be the position of path Pfrom time step n−1 and P_(vx)[n−1]. P_(vy)[n−1] the last known velocity,the velocity-predicted path continuation is then:P _(predx) [n]=P _(x) [n−1]+ΔtP _(vx) [n−1]  (24)P _(predy) [n]=P _(y) [n−1]+ΔtP _(vy) [n−1]  (25)

Letting the set of paths active in the previous image be PA, and let theset electrode groups constructed in the current image be G, step 322finds for each group Gk the closest active path and records the distanceto it:

$\begin{matrix}{\begin{matrix}{{Gk}_{closestP} = {\arg\begin{matrix}\min \\{{Pl} \in {PA}}\end{matrix}_{\;}{d^{2}\left( {{Gk},\;{Pl}} \right)}}} & {\forall{{Gk}\; \in G}}\end{matrix}} & (26) \\\begin{matrix}{{Gk}_{{{closestPdist}^{2}}^{\;}} = {\begin{matrix}\min \\{{Pl} \in {PA}}\end{matrix}{d^{2}\left( {{Gk},\;{Pl}} \right)}}} & {\forall{{Gk}\; \in G}}\end{matrix} & (27)\end{matrix}$where the squared Euclidean distance is an easily computed distancemetric:d ²(Gk,Pl)=(Gk _(x) −Pl _(predx))²+(Gk _(y) −Pl _(predy))²  (28)

Step 324 then finds for each active path Pl, the closest active groupand records the distance to it:

$\begin{matrix}{\begin{matrix}{{Pl}_{closestG} = {\arg\begin{matrix}\min \\{{Gk} \in G}\end{matrix}_{\;}{d^{2}\left( {{Gk},\;{Pl}} \right)}}} & {\forall{{Pl}\; \in {PA}}}\end{matrix}} & (29) \\\begin{matrix}{{Pl}_{{{closestGdist}^{2}}^{\;}} = {\begin{matrix}\min \\{{Gk} \in G}\end{matrix}{d^{2}\left( {{Gk},\;{Pl}} \right)}}} & {\forall{{Pl}\; \in {PA}}}\end{matrix} & (30)\end{matrix}$

In step 326, an active group Gk and path Pl are only paired with oneanother if they are closest to one another, i.e., Gk_(closestP) andPl_(closestG) refer to one another, and the distance between them isless than the tracking radius. All of the following conditions musthold:Gk_(closestP)≡Pl  (31)Pl_(closestG)≡Gk  (32)Pl_(closestGdist2)<Pl_(mack2)  (33)

To aid in detection of repetitive taps of the same finger, it may beuseful to preserve continuity of path assignment between taps over thesame location. This is accomplished in step 334 by repeating steps322-326 using only groups which were left unpaired above and paths whichwere deactivated within the last second or so due to finger liftoff.

In step 336, any group which has still not be paired with an active orrecently deactivated path is allocated a new path, representingtouchdown of a new finger onto the surface. In step 344, any active pathwhich cannot be so paired with a group is deactivated, representing handpart liftoff from the surface.

Step 346 incorporates the extracted parameters of each group into itsassigned path via standard filtering techniques. The equations shownbelow apply simple autoregressive filters to update the path position(P_(x)[n],P_(y)[n],P_(z)[n]), velocity (P_(x)[n],P_(y)[n]), and shape(P_(θ)[n],P_(∈)[n]) parameters from corresponding group parameters, butKalman or finite impulse response filters would also be appropriate.

If a path P has just been started by group G at time step n, i.e., ahand part has just touched down, its parameters are initialized asfollows:P _(x) [n]=G _(x)  (34)P _(y) [n]=G _(y)  (35)P _(z) [n]=G _(z)  (36)P _(θ) [n]=G _(θ)  (37)P _(∈) [n]=G _(∈)  (38)P _(vx) [n]=0  (39)P _(vy) [n]=0  (40)P _(vz) [n]=G _(z) /Δt  (41)else if group G is a continuation of active path P[n−1] to time step n:P _(x) [n]=G _(α) G _(x)+(1−G _(α))(P _(predx) [n−1])  (42)P _(y) [n]=G _(α) G _(y)+(1−G _(α))(P _(predy) [n−1])  (43)P _(z) [n]=G _(α) G _(z)+(1−G _(α))(P _(predz) [n−1])  (44)P _(θ) [n]=G _(α) G _(θ)+(1−G _(α))(P _(θ) [n−1])  (45)P _(∈) [n]=G _(α) G _(∈)+(1−G _(α))(P _(∈) [n−1])  (46)P _(vx) [n]=(P _(x) [n]−P _(x) [n−1])/Δt  (47)P _(vy) [n]=(P _(y) [n]−P _(y) [n−1])/Δt  (48)P _(vz) [n]=(P _(z) [n]−P _(z) [n−1])/Δt  (49)

It is also useful to compute the magnitude P_(speed) and angle P_(dir)from the velocity vector (P_(vx), P_(vx)). Since the reliability ofposition measurements increases considerably with total proximity P_(z),the low-pass filter pole G_(α) is decreased for groups with totalproximities lower than normal. Thus when signals are weak, the systemrelies heavily on the previously established path velocity, but when thefinger firmly touches the surface causing a strong, reliable signal, thesystem relies entirely on the current group centroid measurement.

The next process within the tracking module is contact identification.On surfaces large enough for multiple hands, the contacts of each handtend to form a circular cluster, and the clusters tend to remainseparate because users like to avoid entangling the fingers of oppositehands. Because the arrangement of fingers within a hand cluster isindependent of the location of and arrangement within the other hand'scluster, the contact identification system is hierarchically split. Thehand identification process 247 first decides to which cluster eachcontact belongs. Then a within-cluster identification process 248analyzes for each hand the arrangement of contacts within the hand'scluster, independent of the other hand's cluster. Because within-clusteror finger identification works the same for each hand regardless of howmany hands can fit on the surface, it will be described first. Thedescription below is for identification within the right hand. Mirrorsymmetry must be applied to some parameters before identifying left handcontacts.

FIG. 23 shows the preferred embodiment of the finger identificationprocess 248. For the contacts assigned to each hand this embodimentattempts to match contacts to a template of hand part attractor points,each attractor point having an identity which corresponds to aparticular finger or palm heel. This matching between contact paths andattractors should be basically one to one but in the case that some handparts are not touching the surface, some attractors will be leftunfilled, i.e., assigned to the null path or dummy paths.

Step 350 initializes the locations of the attractor points to theapproximate positions of the corresponding fingers and palms when thehand is in a neutral posture with fingers partially curled. Preferablythese are the same default finger locations (Fi_(defx),Fi_(defy))employed in hand offset estimation. Setting the distances and anglesbetween attractor points from a half-closed hand posture allows thematching algorithm to perform well for a wide variety of finger flexionsand extensions.

The resulting attractor points tend to lie in a ring as displayed by thecrosses in FIG. 24. The identities of attractor points 371-377correspond to the identities of hand parts 201-207. If the given hand isa left hand, the attractor ring must be mirrored about the vertical axisfrom that shown. FIG. 24 also includes line segments 380 forming theVoronoi cell around each attractor point. Every point within anattractor's Voronoi cell is closer to that attractor than any otherattractor. When there is only one contact in the cluster and itsfeatures are not distinguishing, the assignment algorithm effectivelyassigns the contact to the attractor point of the Voronoi cell which thecontact lies within. When there are multiple surface contacts in a handcluster, they could all lie in the same Voronoi cell, so the assignmentalgorithm must perform a global optimization which takes into accountall of the contact positions at once.

Alternative embodiments can include additional attractors for other handpart or alternative attractor arrangements for atypical handconfigurations. For example, attractors for forepalm contacts can beplaced at the center of the ring, but since the forepalms typically donot touch the surface unless the rest of the hand is flattened onto thesurface as well, forepalm attractors should be weighted such thatcontacts are assigned to them only when no regular attractors are leftunassigned.

For optimal matching accuracy the ring should be kept roughly centeredon the hand cluster. Therefore step 352 translates all of the attractorpoints for a given hand by the hand's estimated position offset. Thefinal attractor positions (Aj_(x)[n],Aj_(y)[n]) are therefore given by:Aj _(x) [n]=H _(eox) [n]+Fj _(defx)  (50)Aj _(y) [n]=H _(eoy) [n]+Fj _(defy)  (51)

In alternative embodiments the attractor ring can also be rotated orscaled by estimates of hand rotation and size such as the estimatedfinger offsets, but care must be taken that wrong finger offsetestimates and identification errors do not reinforce one another byseverely warping the attractor ring.

Once the attractor template is in place, step 354 constructs a squarematrix [d_(ij)] of the distances in the surface plane from each activecontact path Pi to each attractor point Aj. If there are fewer surfacecontacts than attractors, the null path P0, which has zero distance toeach attractor, takes place of the missing contacts. Though any distancemetric can be used, the squared Euclidean distance,d _(ij)=(Aj _(x) [n]−Pi _(x) [n])²+(Aj _(y) [n]−Pi _(y) [n])  (52)is preferred because it specially favors assignments wherein the anglebetween any pair of contacts is close to the angle between the pair ofattractors assigned to those contacts. This corresponds to thebiomechanical constraint that fingertips avoid crossing over oneanother, especially while touching a surface.

In step 356, the distances from each contact to selected attractors areweighted according to whether the geometrical features of the givencontact match those expected from the hand part that the attractorrepresents. Since the thumb and palm heels exhibit the mostdistinguishing geometrical features, weighting functions are computedfor the thumb and palm heel attractors, and distances to fingertipattractors are unchanged. In a preferred embodiment, each weightingfunction is composed of several factor versus feature relationships suchas those plotted approximately in FIG. 25. Each factor is designed totake on a default value of 1 when its feature measurement provides nodistinguishing information, take on larger values if the measuredcontact feature uniquely resembles the given thumb or palm hand part,and take on smaller values if the measured feature is inconsistent withthe given attractor's hand part. The factor relationships can bevariously stored and computed as lookup tables, piecewise linearfunctions, polynomials, trigonometric functions, rational functions, orany combination of these. Since assignment between a contact and anattractor whose features match is favored as the weighted distancebetween becomes smaller, the distances are actually weighted(multiplied) with the reciprocals of the factor relationships shown.

FIG. 25A shows the right thumb and right inner palm heel orientationfactor versus orientation of a contact's fitted ellipse. Orientation ofthese hand parts tends to be about 120°, whereas fingertip and outerpalm heel contacts are usually very close to vertical (90°), andorientation of the left thumb and left inner palm heel averages 60°. Theright orientation factor therefore approaches a maximum at 120°. Itapproaches the default value of 1 at 0°, 90°, and 180° where orientationis inconclusive of identity, and reaches a minimum at 60°, the favoredorientation of the opposite thumb or palm heel. The correspondingrelationship for the left thumb and inner palm heel orientation factoris flipped about 90°.

FIG. 25B approximately plots the thumb size factor. Since thumb size asindicated by total proximity tends to peak at two or three times thesize of the typical curled fingertip, the thumb size factor peaks atthese sizes. Unlike palm heels, thumb contacts can not be much largerthan two or three times the default fingertip size, so the thumb factordrops back down for larger sizes. Since any hand part can appear smallwhen touching the surface very lightly or just starting to touchdown,small size is not distinguishing, so the size factor defaults to 1 forvery small contacts.

FIG. 25C approximately plots the palm heel size factor. As more pressureis applied to the palms, the palm heel contacts can grow quite large,remaining fairly round as they do so. Thus the palm heel size factor ismuch like the thumb size factor except the palm factor is free toincrease indefinitely. However, fingertip contacts can grow by becomingtaller as the fingers are flattened. But since finger width is constant,the eccentricity of an ellipse fitted to a growing fingertip contactincreases in proportion to the height. To prevent flattened fingers fromhaving a large palm factor, has little effect for palms, whoseeccentricity remains near 1, but cancels the high proximities offlattened fingertips. Though directly using fitted ellipse width wouldbe less accurate for low resolution electrode arrays, the above ratiobasically captures contact width.

Another important distinguishing feature of the palm heels is that wristanatomy keeps the centroids of their contacts separated from one otherand from the fingers by several centimeters. This is not true of thethumb and fingertips, which can be moved within a centimeter of oneanother via flexible joints. The inter-palm separation feature ismeasured by searching for the nearest neighbor contact of a givencontact and measuring the distance to the neighbor. As plottedapproximately in FIG. 25D, the palm separation factor quickly decreasesas the separation between the contact and its nearest neighbor fallsbelow a few centimeters, indicating that the given contact (and itsnearest neighbor) are not palm heels. Unlike the size and orientationfactors which only become reliable as the weight of the hands fullycompresses the palms, the palm separation factor is especially helpfulin distinguishing the palm heels from pairs of adjacent fingertipsbecause it applies equally well to light, small contacts.

Once the thumb and palm weightings have been applied to the distancematrix, step 358 finds the one-to-one assignment between attractors andcontacts which minimizes the sum of weighted; distances between eachattractor and it's assigned contact. For notational purposes, let a newmatrix [c_(ij)] hold the weighted distances:

$\begin{matrix}{c_{ij} = \left\{ \begin{matrix}{d_{ij}/\left( {{Pi}_{{thumb\_ size}{\_ fact}}{Pi}_{orient\_ fact}} \right)} & {{{if}\mspace{14mu} j} = 1} \\d_{ij} & {{{if}\mspace{14mu} 2} \leq j < 5} \\{d_{ij}/\left( {{Pi}_{{palm\_ size}{\_ fact}}{Pi}_{{palm\_ sep}{\_ fact}}} \right)} & {{{if}\mspace{14mu} j} = 6} \\{d_{ij}/\left( {{Pi}_{{palm\_ size}{\_ fact}}{Pi}_{{palm\_ sep}{\_ fact}}} \right)} & {{{if}\mspace{14mu} j} = 7}\end{matrix} \right.} & (53)\end{matrix}$Mathematically the optimization can then be stated as finding thepermutation {π₁, . . . , π₇} of integer hand part identities {1, . . . ,7} which minimizes:

$\begin{matrix}{\sum\limits_{i = 1}^{7}c_{i\;\pi_{i}}} & (54)\end{matrix}$where c_(ij) is the weighted distance from contact i to attractor j, andcontact i and attractor j are considered assigned to one another whenπ_(i)≡j. This combinatorial optimization problem, known morespecifically in mathematics as an assignment problem, can be efficientlysolved by a variety of well-known mathematical techniques, such asbranch and bound, localized combinatorial search, the Hungarian method,or network flow solvers. Those skilled in the art will recognize thatthis type of combinatorial optimization problem has a mathematicallyequivalent dual representation in which the optimization is reformulatedas a maximization of a sum of dual parameters. Such reformulation of theabove hand part identification method as the dual of attractor-contactdistance minimization remains well within the scope of this invention.

To avoid unnecessary computation, decision diamond 360 ends the fingeridentification process at this stage if the hand assignment of the givencontact cluster is only a tentative hypothesis being evaluated by thehand identification module 247. However, if the given hand assignmentsare the final preferred hypothesis, further processes verify fingeridentities and compile identity statistics such as finger counts.

The identifications produced by this attractor assignment method arehighly reliable when all five fingers are touching the surface or whenthumb and palm features are unambiguous. Checking that the horizontalcoordinates for identified fingertip contacts are in increasing ordereasily verifies that fingertip identities are not erroneously swapped.However, when-only two to four fingers are touching, yet no fingerstrongly exhibits thumb size or orientation features, the assignment ofthe innermost finger contact may wrongly indicate whether the contact isthe thumb. In this case, decision diamond 362 employs a thumbverification process 368 to take further measurements between theinnermost finger contact and the other fingers. If these furthermeasurements strongly suggest the innermost finger contact identity iswrong, the thumb verification process changes the assignment of theinnermost finger contact. Once the finger assignments are verified, step364 compiles statistics about the assignments within each hand such asthe number of touching fingertips and bitfields of touching fingeridentities. These statistics provide convenient summaries ofidentification results for other modules.

FIG. 26 shows the steps within the thumb verification module. The first400 is to compute several velocity, separation, and angle factors forthe innermost contact identified as a finger relative to the othercontacts identified as fingers. Since these inter-path measurementspresuppose a contact identity ordering, they could not have easily beenincluded as attractor distance weightings because contact identities arenot known until the attractor distance minimization is complete. For thefactor descriptions below, let FI be the innermost finger contact, FN bethe next innermost finger contact, FO be the outermost finger contact.

The separation between thumb and index finger is often larger than theseparations between fingertips, but all separations tend to grow as thefingers are outstretched. Therefore an inner separation factorinner_separation_fact is defined as the ratio of the distance betweenthe innermost and next innermost finger contacts to the average of thedistances between other adjacent fingertip contacts, avg_separation: 12innerseparationfact min

$\begin{matrix}{{innerseparationfact} \approx {\min\left( {1,\frac{\sqrt{\begin{matrix}{\left( \;{{FI}_{\; x}\; - \;{FN}_{\; x}} \right)^{2} +} \\\left( \mspace{11mu}{{FI}_{\; y}\mspace{11mu} - \mspace{14mu}{FN}_{\; y}} \right)^{2}\end{matrix}}}{avgseparation}} \right)}} & (55)\end{matrix}$

The factor is clipped to be greater than one since an innermostseparation less than the average can occur regardless of whether thumbor index finger is the innermost touching finger. In case there are onlytwo finger contacts, a default average separation of 2-3 cm is used. Thefactor tends to become larger than one if the innermost contact isactually the thumb but remains near one if the innermost contact is afingertip.

Since the thumb rarely moves further forward than the fingertips exceptwhen the fingers are curled into a fist, the angle between the innermostand next innermost finger contact can help indicate whether theinnermost finger contact is the thumb. For the right hand the angle ofthe vector from the thumb to the index finger is most often 60°, thoughit ranges to 0° as the thumb moves forward and to 120° as the thumbadducts under the palm. This is reflected in the approximate plot of theinner angle factor in FIG. 32, which peaks at 60° and approaches 0toward 0° and 120°. If the innermost finger contact is actually an indexfingertip, the measured angle between innermost and next innermostcontact will likely be between 30° and −60°, producing a very smallangle factor.

The inner separation and angle factors are highly discriminating ofneutral thumb postures, but users often exceed the above citedseparation and angle ranges when performing hand scaling or rotationgestures. For instance, during an anti-pinch gesture, the thumb maystart pinched against the index or middle fingertip, but then the thumband fingertip slide away from one another. This causes the innerseparation factor to be relatively small at the start of the gesture.Similarly, the thumb-index angle can also exceed the range expected bythe inner angle factor at the beginning or end of hand rotationgestures, wherein the fingers rotate as if turning a screw. Tocompensate, the inner separation and angle factors are fuzzy OR'ed withexpansion and rotation factors which are selective for symmetric fingerscalings or rotations centered on a point between the thumb andfingertips.

When defined by the following approximate equation, the expansion factorpeaks as the innermost and outermost finger contacts slide atapproximately the same speed and in opposite directions, parallel to thevector between them:

$\begin{matrix}\begin{matrix}{{expansionfact} \approx {{- \sqrt{{{FI}_{speed}\lbrack n\rbrack} \times {{FO}_{speed}\lbrack n\rbrack}}} \times}} \\{{\cos\left( {{{FI}_{dir}\lbrack n\rbrack} - {\angle\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)}} \right)} \times} \\{\cos\left( {{{FO}_{dir}\lbrack n\rbrack} - {\angle\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)}} \right)}\end{matrix} & (56) \\{{{{expansnsion\_ fact}\text{:}} = {\max\left( {0,{expansion\_ fact}} \right)}}{{where}\mspace{14mu}{\angle\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)}\mspace{14mu}{is}\mspace{14mu}{the}\mspace{14mu}{angle}\mspace{14mu}{between}\mspace{14mu}{the}\mspace{14mu}{fingers}\text{:}}} & (57) \\{{\angle\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)} = {\arctan\left( \frac{{{FI}_{y}\lbrack n\rbrack} - {{FO}_{y}\lbrack n\rbrack}}{{{FI}_{x}\lbrack n\rbrack} - {{FO}_{x}\lbrack n\rbrack}} \right)}} & (58)\end{matrix}$

Translational motions of both fingers in the same direction producenegative factor values which are clipped to zero by the max operation.Computing the geometric rather than arithmetic mean of the innermost andoutermost speeds aids selectivity by producing a large expansion factoronly when speeds of both contacts are high.

The rotation factor must also be very selective. If the rotation factorwas simply proportional to changes in the angle between innermost andoutermost finger, it would erroneously grow in response to asymmetriesin finger motion such as when the innermost finger starts translatingdownward while the outermost contact is stationary. To be moreselective, the rotation factor must favor symmetric rotation about animaginary pivot between the thumb and fingertips. The approximaterotation factor equation below peaks as the innermost and outermostfinger move in opposite directions, but in this case the contacts shouldmove perpendicular to the vector between them:

$\begin{matrix}\begin{matrix}{{rotationfact} = {{- \sqrt{{{FI}_{speed}\lbrack n\rbrack} \times {{FO}_{speed}\lbrack n\rbrack}}} \times}} \\{{\sin\left( {{{FI}_{dir}\lbrack n\rbrack} - {\angle\;\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)}} \right)} \times} \\{\sin\left( {{{FO}_{dir}\lbrack n\rbrack} - {\angle\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)}} \right)}\end{matrix} & (59) \\{{rotation\_ fact}\text{:} = {\max\left( {0,{rotation\_ fact}} \right)}} & (60)\end{matrix}$

Since motions which maximize this rotation factor are easy to performbetween the opposable thumb and another finger but difficult to performbetween two fingertips the rotation factor is a robust indicator ofthumb presence.

Finally, a fuzzy logic expression (step 402) combines theseinter-contact factors with the thumb feature factors for the innermostand next innermost finger contacts. In a preferred embodiment, thisfuzzy logic expression for the combined_thumb_fact takes the form:combined_thumb_fact≈(inner_separation_fact×angle_fact+expansion_fact+rotation_fact)×(FI_(orient) _(—) _(fact) /FN _(orient) _(—) _(fact))×(FI _(thumb) _(—)_(size) _(—) _(fact) /FN _(thumb) _(—) _(size) _(—) _(fact))

The feature factor ratios of this expression attempt to compare thefeatures of the innermost contact to current features of the nextinnermost contact, which is already known to be a fingertip. If theinnermost contact is also a fingertip its features should be similar tothe next innermost, causing the ratios to remain near one. However,thumb-like features on the innermost contact will cause the ratios to belarge. Therefore if the combined thumb factor exceeds a high threshold,diamond 404 decides the innermost finger contact is definitely a thumb.If decision diamond 412 determines the contact is not already assignedto the thumb attractor 412, step 414 shifts the contact assignmentinward on the attractor ring to the thumb attractor. Otherwise, ifdecision diamond 406 determines that the combined thumb factor is lessthan a low threshold, the innermost contact is most definitely not thethumb. Therefore if decision diamond 408 finds the contact assigned tothe thumb attractor, step 410 shifts the innermost contact assignmentand any adjacent finger contacts outward on the attractor ring to unfillthe thumb attractor. If the combined_thumb_fact is between the high andlow threshold or if the existing assignments agree with the thresholddecisions, step 413 makes no assignment changes.

The hand contact features and interrelationships introduced here to aididentification can be measured and combined in various alternative waysyet remain well within the scope of the invention. In alternativeembodiments of the multi-touch surface apparatus which include raised,touch-insensitive palm rests, palm identification and its requisiteattractors and factors may be eliminated. Geometrical parameters can beoptimally adapted to measurements of individual user hand size takenwhile the hand is flattened. However, the attractor-based identificationmethod already tolerates variations in a single person's fingerpositions due to finger flexion and extension which are as great orgreater than the variations in hand size across adult persons. Thereforeadaptation of the thumb and palm size factors to a person's averagefinger and palm heel proximities is more important than adaptation ofattractor positions to individual finger lengths, which will only addmarginal performance improvements.

As another example of an alternative method for incorporating thesefeatures and relationships into a hand contact identifier, FIG. 27diagrams an alternative finger identification embodiment which does notinclude an attractor template. To order the paths from finger and palmcontacts within a given hand 430, step 432 constructs a two-dimensionalmatrix of the distances from each contact to the other contacts. In step434, a shortest path algorithm well known from the theory of networkflow optimization then finds the shortest graph cycle connecting all thecontact paths and passing through each once 434. Since hand contactstend to lie in a ring this shortest graph cycle will tend to connectadjacent contacts, thus establishing a sensible ordering for them.

The next step 438 is to pick a contact at an extreme position in thering such as the innermost or outermost and test whether it is a thumb(decision diamond 440) or palm (decision diamond 442). This can be doneusing contact features and fuzzy logic expressions analogous to thoseutilized in the thumb verification process and the, attractorweightings. If the innermost path is a thumb, step 444 concludes thatcontacts above are most likely fingertips, and contacts in the ringbelow the thumb are most likely palms. If (442) the innermost path is apalm heel, step 446 concludes the paths significantly above theinnermost must be fingers while paths at the same vertical level shouldbe palms. The thumb and palm tests are then repeated for the contactsadjacent in the ring to the innermost until any other thumb or palmcontacts are found. Once any thumb and palm contacts are identified,step 448 identifies remaining fingertip contacts by their respectiveordering in the ring and their relatively high vertical position.

Since this alternative algorithm does not include an attractor templateto impose constraints on relative positions, the fuzzy verificationfunctions for each contact may need to include measurements of thevertical position of the contact relative to other contacts in the ringand relative to the estimated hand offset. The attractor templateembodiment is preferred over this alternative embodiment because theattractor embodiment more elegantly incorporates expected angles betweencontacts and the estimated hand offset into the finger identificationprocess.

Hand identification is needed for multi-touch surfaces which are largeenough to accomodate both hands simultaneously and which have the leftand right halves of the surface joined such that a hand can roam freelyacross the middle to either half of the surface. The simplest method ofhand identification would be to assign hand identity to each contactaccording to whether the contact initially touched down in the left orright half of the surface. However, if a hand touched down in themiddle, straddling the left and right halves, some of the hand'scontacts would end up assigned to the left hand and others to the righthand. Therefore more sophisticated methods which take into account theclustering properties of hand contacts must be applied to ensure allcontacts from the same hand get the same identity. Once all surfacecontacts are initially identified, the path tracking module can reliablyretain existing identifications as a hand slides from one side of thesurface to the other.

The thumb and inner palm contact orientations and the relative thumbplacement are the only contact features independent of cluster positionwhich distinguish a lone cluster of right hand contacts from a clusterof left hand contacts. If the thumb is lifted off the surface, a righthand contact cluster appears nearly indistinguishable from a left handcluster. In this case cluster identification must still depend heavilyon which side of the board the cluster starts on, but the identity ofcontacts which recently lifted off nearby also proves helpful. Forexample, if the right hand moves from the right side to the middle ofthe surface and lifts off, the next contacts which appear in the middlewill most likely be from the right hand touching back down, not from theleft hand moving to the middle and displacing the right hand. Thedivision between left and right halves of the surface should thereforebe dynamic, shifting toward the right or left according to which handwas most recently near the middle. Since the hand offset estimatestemporarily retain the last known hand positions after liftoff, such adynamic division is implemented by tying the positions of left hand andright hand attractor templates to the estimated hard positions.

Though cases remain in which the user can fool the hand identificationsystem with sudden placements of a hand in unexpected locations, theuser may actually wish to fool the system in these cases. For example,users with only one hand free to use the surface may intentionally placethe hand far onto the opposite half of the surface to access the chordinput operations of the opposite hand. Therefore, when a hand clustersuddenly touches down well into the opposite half of the surface, it cansafely be given the opposite halfs identity, regardless of its trueidentity. Arching the surface across the middle can also discourageusers from sliding a hand to the opposite side by causing awkwardforearm pronation should users do so.

FIG. 29 shows process details within the hand identification module 247.Decision diamond 450 first determines whether the hand identificationalgorithm actually needs to be executed by checking whether all pathproximities have stabilized. To maximize stability of theidentifications, hand and finger identities need only be reevaluatedwhen a new hand part touches down or disambiguating features of existingcontacts become stronger. The contact size and orientation features areunreliable until the flesh fully compresses against the surface a fewdozen milliseconds after initial surface contact. Therefore decisiondiamond 450 executes the hand identification algorithm for eachproximity image in which a new contact appears and for subsequentproximity images in which the total proximity of any new contactscontinues to increase. For images in which proximities of existingcontacts have stabilized and no new contacts appear, path continuationas performed by the path tracking process 245 is sufficient to retainand extend (step 452) the contact identifications computed from previousimages.

Should the hand identification algorithm be invoked for the currentimage, the first step 453 is to define and position left and right handattractor templates. These should be basically the same as the attractortemplates (FIG. 24, step 352) used in within-hand identification, exceptthat both left and right rings must now be utilized at once. The defaultplacement of the rings relative to one another should correspond to thedefault left and right hand contact positions shown in FIG. 20A. Eachring translates to follow the estimated position of its hand, just likethe sloppy segmentation regions follow the hands in FIG. 20B. Individualattractor points can safely be translated by their correspondingestimated finger offsets. Therefore the final attractor positions(Aj_(x)[n],Aj_(y)[n]) for the left hand L and right hand H attractorrings are:Laj_(x) [n]=Lh _(eox) [n]+LFj _(eox) [n]+Lfj _(defx)  (62)Laj _(x) [n]=Lh _(eoy) [n]+LFj _(eoy) [n]+Lfj _(defy)  (63)Raj _(x) [n]=Rh _(eox) [n]+RFj _(eox) [n]+Rfj _(defx)  (64)Raj _(y) [n]=Rh _(eoy) [n]+RFj _(eoy) [n]+Rfj _(defy)  (65)

Basically the hand identification algorithm will compare the cost ofassigning contacts to attractors in one ring versus the other, the costdepending on the sum of weighted distances between each contact and itsassigned attractor. Adjusting the attractor ring with the estimated handand finger offsets lowers the relative costs for assignment hypotheseswhich resemble recent hand assignments, helping to stabilizeidentifications across successive proximity images even when handstemporarily lift off.

Next a set of assignment hypotheses must be generated and compared. Themost efficient way to generate sensible hypotheses is to define a set ofroughly vertical contour lines, one between each horizontally adjacentcontact. Step 454 does this by ordering all surface contacts by theirhorizontal coordinates and establishing a vertical contour halfwaybetween each pair of adjacent horizontal coordinates. FIGS. 30A-C showexamples of three different contours 475 and their associated assignmenthypotheses for a fixed set of contacts. Each contour corresponds to aseparate hypothesis, known also as a partition, in which all contacts tothe left 476 of the contour are from the left hand, and all contacts tothe right 477 of the contour are from the right hand. Contours are alsonecessary at the left and right ends of the surface to handle thehypotheses that all contacts on the surface are from the same hand.Contours which hypothesize more contacts on a given hand than can becaused by a single hand are immediately eliminated.

Generating partitions via vertical contours avoids all hypotheses inwhich contacts of one hand horizontally overlap or cross over contactsof the opposite hand. Considering that each hand can cause seven or moredistinct contacts, this reduces the number of hand identity permutationsto examine from thousands to at most a dozen. With fewer hypotheses toexamine, the evaluation of each partition can be much moresophisticated, and if necessary, computationally costly.

The optimization search loop follows. Its goal is to determine which ofthe contours divides the contacts into a partition of two contactclusters such that the cluster positions and arrangement of contactswithin each cluster best satisfy known anatomical and biomechanicalconstraints. The optimization begins by picking (step 456) a firstcontour divider such as the leftmost and tentatively assigning (step458) any contacts to the left of the contour to the left hand and therest to the right hand. Step 460 invokes the finger identificationalgorithm of FIG. 23, which attempts to assign finger and palmidentities to contacts within each hand. Decision diamond 360 avoids thecomputational expense of thumb verification 368 and statistics gathering364 for this tentative assignment hypothesis.

Returning to FIG. 29, step 462 computes a cost for the partition. Thiscost is meant to evaluate how well the tentatively identified contactsfit their assigned attractor ring and how well the partition meetsbetween-hand separation constraints. This is done by computing for eachhand the sum of weighted distances from each tentatively identifiedcontact to its assigned attractor point as in Equation 54 of fingeridentification, including size and orientation feature factors for thumband palm attractors. This sum represents the basic template fitting costfor a hand. Each hand cost is then weighted as a whole with thereciprocals of its clutching velocity, handedness, and palm cohesionfactors. These factors, to be described below, represent additionalconstraints which are underemphasized by the weighted attractordistances. Finally, the weighted left and right hand costs are addedtogether and scaled by the reciprocal of a hand separation factor toobtain a total cost for the partition.

If decision diamond 464 determines this total cost is lower than thetotal costs of the partitions evaluated so far 464, step 466 records thepartition cost as the lowest and records the dividing contour. Decisiondiamond 472 repeats this process for each contour 470 until the costs ofall partitions have been evaluated. Step 473 chooses the partition whichhas the lowest cost overall as the actual hand partitioning 473, and thehand identities of all contact paths are updated accordingly. Then step474 reinvokes the within-hand finger contact identification process sothat the thumb verification and statistics gathering processes areperformed using the actual hand assignments.

Users often perform clutching motions in which the right hand, forexample, lifts off from a slide at the right side of the surface,touches backdown in the middle of the surface, and resumes slidingtoward the right. Therefore when a hand is detected touching down in themiddle of the surface and sliding toward one side, it probably came fromthe at side. A hand velocity factor, plotted approximately in FIG. 31A,captures this phenomenon by slightly increasing in value when a handcluster's contacts are moving toward the cluster's assigned side of theboard, thus decreasing the basic cost of the hand. The factor is afunction of the average of the contacts' horizontal velocities the sideof the surface the given cluster is assigned. Since high speeds do notnecessarily give a stronger indication of user intent the factorsaturates at moderate speeds.

Though the thumb orientation factors help identify which hand a thumb isfrom when the thumb lies in the ambiguous middle region of the surface,the vertical position of the thumb relative to other fingers in the samehand also gives a strong indication of handedness. The thumb tends to bepositioned much lower than the fingertips, but the pinky tends to beonly slightly lower than the other fingertips. The handedness factorplotted approximately in FIG. 31B, takes advantage of this constraint byboosting the hand cost when the contact identified as the outermostfingertip is more than a couple centimeters lower than the nextoutermost fingertip contact. In such cases the tentative hand assignmentfor all contacts in the cluster is probably wrong. Since this causes thewithin-hand identification algorithm to fit the contacts to the wrongattractor ring, finger identities become reversed such that thesupposedly lowered pinky is truly a lowered thumb of the opposite hand.Unfortunately, limited confidence can be placed in the handednessfactor. Though the pinky should not appear lowered as much as the thumbthe outer palm heel can, creating an ambiguity in which the thumb andfingertips of one hand have the same contact arrangement as thefingertips and outer palm heel of the opposite hand. This ambiguity cancause the handedness factor to be erroneously low for an accuratelyidentified hand cluster, so the handedness factor is only used onclusters in the middle of the surface where hand position is ambiguous.

Distinguishing contact clusters is challenging because a cluster canbecome quite sparse and large when the fingers outstretched, with thepinky and thumb of the same hand spanning up to 20 cm. However, the palmcan stretch very little in comparison, placing useful constraints on howfar apart palm heel contacts and forepalms from the same hand can be.The entire palm region of an outstretched adult hand is about 10 cmsquare, so palm contact centroids should not be scattered over a regionlarger than about 8 cm. When a partition wrongly includes fingers fromthe opposite hand in a cluster, the within-cluster identificationalgorithm tends to assign the extra fingers from the opposite hand topalm heel and forepalm attractors. This usually causes the contactsassigned to the cluster's palm attractors to be scattered across thesurface wider than is plausible for true palm contacts from a singlehand. To punish such partitions, the palm cohesion factor quickly dropsbelow one for a tentative hand cluster in which the supposed palmcontacts are scattered over a region larger than 8 cm. Therefore itsreciprocal will greatly increase the hand's basic cost. FIG. 31C showsthe value of the palm cohesion factor versus horizontal separationbetween palm contacts. The horizontal spread can be efficiently measuredby finding the maximum and minimum horizontal coordinates of allcontacts identified as palm heels or forepalms and taking the differencebetween the maximum and minimum. The measurement and factor value lookupare repeated for the vertical separation, and the horizontal andvertical factors are multiplicatively combined to obtain the final palmcohesion factor.

FIG. 33 is an approximate plot of the inter-hand separation factor. Thisfactor increases the total costs of partitions in which the estimated oractual horizontal positions of the thumbs from each hand approach oroverlap. It is measured by finding the minimum of the horizontal offsetsof right hand contacts with respect to their corresponding defaultfinger positions. Similarly the maximum of the horizontal offsets of theleft hand contacts with respect to their corresponding default fingerpositions is found. If the difference between these hand offset extremesis small enough to suggest the thumbs are overlapping the same columnarregion of the surface while either touching the surface or floatingabove it, the separation factor becomes very small. Such overlapcorresponds to a negative thumb separation in the plot. To encourageassignment of contacts which are within a couple centimeters of oneanother to the same cluster, the separation factor gradually begins todrop starting with positive separations of a few centimeters or less.The inter-hand separation factor is not applicable to partitions inwhich all surface contacts are assigned to the same hand, and takes onthe default value of one in this case.

Alternative embodiments of this hand identification process can includeadditional constraint factors and remain well within the scope of thisinvention. For example, a velocity coherence factor could be computed tofavor partitions in which all fingers within a cluster slide atapproximately the same speed and direction, though each cluster as awhole has a different average speed and direction.

Sometimes irreversible decisions made by the chord motion recognizer ortyping recognized on the basis of existing hand identifications preventlate changes in the identifications of hand contacts even when newproximity image information suggests existing identifications are wrong.This might be the case for a chord slide which generates input eventsthat can not be undone, yet well into the slide new image informationindicates some fingers in the chord should have been attributed to theopposite hand. In this case the user can be warned to stop the slide andcheck for possible input errors but in the meantime it is best to retainthe existing identifications even if wrong, rather than switch tocorrect assignments which could have further unpredictable effects whenadded to the erroneous input events. Therefore once a chord slide hasgenerated input events, the identifications of their existing paths maybe locked so the hand identification algorithm can only swapidentifications of subsequent new contacts.

This hand identification process can be modified for differentlyconfigured multi-touch surfaces and remain well within the scope of thisinvention. For surfaces which are so narrow that thumbs invade oneanother's space or so tall that one hand can lie above another, thecontours need not be straight vertical lines. Additional contours couldweave around candidate overlapping thumbs, or they could beperpendicular to the vector between the estimated hand positions. If thesurface was large enough for more than one user, additional attractorrings would have to be provided for each additional hand, and multiplepartitioning contours would be necessary per hypothesis to partition thesurface into more than two portions. On a surface large enough for onlyone hand it might still be necessary to determine which hand wastouching the surface. Then instead of hypothesizing different contours,the hand identification module would evaluate the hypotheses that eitherthe left hand attractor ring or the right hand attractor ring wascentered on the surface. If the surface was mounted on a pedestal toallow access from all sides, the hand identification module would alsohypothesize various rotations of each attractor ring.

The attractor-based finger identification system 248 will successfullyidentify the individual hand contacts which comprise the pen grip handconfiguration (FIG. 15). However, additional steps are needed todistinguish the unique finger arrangement within the pen grip from thenormal arrangement within the closed hand configuration (FIG. 14). Inthis pen grip arrangement the outer fingers curl under toward the palmsso their knuckles touch the surface and the index, finger juts out aheadof them. The pen grip detection module 17 employs a fuzzy patternrecognition process similar to the thumb verification process to detectthis unique arrangement

An additional problem with handwriting recognition via the pen grip handconfiguration is that the inner gripping fingers and sometimes the wholehand will be picked up between strokes, causing the distinguishingfinger arrangement to temporarily disappear. Therefore the pen griprecognition process must have hysteresis to stay in handwriting modebetween gripping finger lifts. In the preferred embodiment, hysteresisis obtained by temporal filtering of the combined fuzzy decision factorsand by using the estimated finger positions in measurements of fingerarrangement while the actual fingers are lifted off the surface. Theestimated finger positions provide effective hysteresis because theytemporarily retain the unique jutting arrangement before decaying backtoward the normal arched fingertip positions a few seconds afterliftoff.

FIG. 28 shows the steps within the pen grip detection module 17.Decision diamond 485 determines whether all pen grip hand parts aretouching the surface. If not decision diamond 486 causes the estimatedfinger and palm positions to be retrieved for any lifted parts in step487 only if pen grip or handwriting mode is already active. Otherwisethe process exits for lack of enough surface contacts. Thus theestimated finger positions cannot be used to start handwriting mode, butthey can continue it. Step 488 retrieves the measured positions andsizes of fingers and palm heels which are touching the surface.

Step 489 computes a knuckle factor from the outer finger sizes and theirvertical distance from the palm heels which peaks as the outer fingercontacts become larger than normal fingertips and close to the palmheels. Step 490 computes a jutting factor from the difference betweenthe vertical coordinates of the inner and outer fingers which peaks asthe index fingertip juts further out in front of the knuckles. Step 491combines the knuckle and jutting factors in a fuzzy logic expression andaverages the result with previous results via an autoregressive ormoving average filter. Decision diamond 492 continues or starts pen gripmode if the filtered expression result is above a threshold which mayitself be variable to provide additional hysteresis. While in pen gripmode, typing 12 and chord motion recognition 18 are disabled for the pengripping hand.

In pen grip mode, decision diamond 493 determines whether the innergripping fingers are actually touching the surface. If so, step 495generates inking events from the path parameters of the inner fingersand appends them to the outgoing event queue of the host communicationinterface. These inking events can either cause “digital ink” to belaved on the display 24 for drawing or signature capture purposes, orthey can be intercepted by a handwriting recognition system andinterpreted as gestures or language symbols. Handwriting recognitionsystems are well known in the art.

If the inner fingers are lifted, step 494 sends stylus raised events tothe host communication interface to instruct the handwriting recognitionsystem of a break between symbols. In some applications the user mayneed to indicate where the “digital ink” or interpreted symbols are tobe inserted on the display by positioning a cursor. Though on amulti-touch surface a user could move the cursor by leaving the pen gripconfiguration and sliding a finger chord, it is preferable to allowcursor positioning without leaving the pen grip configuration. This canbe supported by generating cursor positioning events from slides of thepalm heels and outer knuckles. Since normal writing motions will alsoinclude slides of the palm heels and outer knuckles, palm motions shouldbe ignored until the inner fingers have been lifted for a few hundredmilliseconds.

Should the user actually pick up a conductive stylus and attempt towrite with it, the hand configuration will change slightly because theinner gripping fingers will be directing the stylus from above thesurface rather than touching the surface during strokes. Since theforearm tends to supinate more when actually holding a stylus, the innerpalm heel may also stay off the surface while the hand rests on thesides of the pinky, ring finger and the outer palm heel. Though theouter palm heel may lie further outward than normal with respect to thepinky, the ring and pinky fingers will still appear as large knucklecontacts curled close to the outer palm. The tip of the stylusessentially takes the place of the index fingertip for identificationpurposes, remaining at or above the vertical level of the knuckles. Thusthe pen grip detector can function in essentially the same way when theuser writes with a stylus, except that the index fingertip path sent tothe host communication interface will in actuality be caused by thestylus.

Technically, each hand has 24 degrees of freedom of movement in allfinger joints combined, but as a practical matter, tendon linkagelimitations make it difficult to move all of the joints independently.Measurements of finger contacts on a surface yield ten degrees offreedom in motion lateral to the surface, five degrees of freedom inindividual fingertip pressure or proximity to the surface, and onedegree of freedom of thumb orientation. However, many of these degreesof freedom have limited ranges and would require unreasonable twistingand dexterity from the average user to access independently.

The purpose of the motion component extraction module 16 is to extractfrom the 16 observable degrees of freedom enough degrees of freedom forcommon graphical manipulation tasks in two and three dimensions. In twodimensions the most common tasks are horizontal and vertical panning,rotating, and zooming or resizing. In three dimensions, two additionalrotational degrees of freedom are available around the horizontal andvertical axes. The motion component extractor attempts to extract these4-6 degrees of freedom from those basic hand motions which can beperformed easily and at the same time without interfering with oneanother. When multiple degrees of freedom can be accessed at the sametime they are said to be integral rather than separable, and integralinput devices are usually faster because they allow diagonal motionsrather than restricting motions to be along a single axis or degree offreedom at one time.

When only four degrees of freedom are needed, the basic motions can bewhole hand translation, hand scaling by uniformly flexing or extendingthe fingers, and hand rotation either about the wrist as when unscrewingajar lid or between the fingers as when unscrewing a nut. Not only arethese hand motions easy to perform because they utilize motions whichintuitively include the opposable thumb, they correspond cognitively tothe graphical manipulation tasks of object rotation and sizing. Theironly drawback is that the translational motions of all the fingersduring these hand rotations and scalings do not cancel perfectly and caninstead add up to a net translation in some direction in addition to thedesired rotation or scaling. To allow all motions to be performedsimultaneously so that the degrees of freedom are integral yet toprevent unintended translations from imperfectly performed scalings androtations, the motion extractor preferentially weights the fingers whosetranslations cancel best and nonlinearly scales velocity componentsdepending on their speeds relative to one another.

The processes within the motion component extractor 16 are shown in FIG.34. Step 500 first fetches the identified contact paths 250 for thegiven hand. These paths contain the lateral velocities and proximitiesto be used in the motion calculations, and the identifications areneeded so that motion of certain fingers or palm heels which woulddegrade particular motion component calculations can be deemphasized.

The next step 502 applies additional filtering to the lateral contactvelocities when finger proximity is changing rapidly. This is necessarybecause during finger liftoff and touch down on the surface, the frontpart of the fingertip often touches down before and lifts off after theback of the fingertip, causing a net downward or upward lateraltranslation in the finger centroid. Such proximity-dependenttranslations can be put to good use when slowly rolling the fingertipfor fine positioning control, but they can also annoy the user if theycause the cursor to jump away from a selected position during fingerliftoff. This is prevented by temporarily downscaling a finger's lateralvelocity in proportion to large changes in the finger's proximity. Sinceother fingers within a hand tend to shift slightly as one finger liftsoff, additional downscaling of each finger velocity is done in responseto the maximum percent change in proximity among contacting fingers.Alternatively, more precise suppression can be obtained by subtractingfrom the lateral finger speed an amount proportional to theinstantaneous change in finger contact height. This assumes that theperturbation in lateral finger velocity caused by finger liftoff isproportional to the change in contact height due to the back of thefingertip lifting off first or touching down last.

Process 504, whose detailed steps are shown in FIG. 36, measures thepolar velocity components from radial (scaling) and rotational motion.Unless rotation is extracted from thumb orientation changes, at leasttwo contacting fingers are necessary to compute a radial or angularvelocity of the hand. Since thumb motion is much more independent of theother fingers than they are of one another, scalings and rotations areeasier for the user to perform if one of these fingers is the opposablethumb, but the measurement method will work without the thumb. Ifdecision diamond 522 determines that less than two fingers are touchingthe surface, step 524 sets the radial and rotational velocities of thehand to zero. FIG. 35 shows trajectories of each finger during acontractive hand scaling. The thumb 201 and pinky 205 travel in nearlyopposite directions at roughly the same speed, so that the sum of theirmotions cancels for zero net translation, but the difference in theirmotions is maximized for a large net scaling. The central fingers202-204 also move toward a central point but the palm heels remainstationary, failing to complement the flexing of the central fingers.Therefore the difference between motion of a central finger and anyother finger is usually less than the difference between the pinky andthumb motions, and the sum of central finger velocities during a handscaling adds up to a net vertical translation. Similar phenomena occurduring hand rotations, except that if the rotation is centered at thewrist with forearm fixed rather than centered at the forepalms, a nethorizontal translation will appear in the sum of motions from anycombination of fingers.

Since the differences in finger motion are usually greatest betweenthumb and pinky, step 526 only retrieves the current and previouspositions of the innermost and outermost touching fingers for the handscaling and rotation measurements.

Step 528 then computes the hand scaling velocity H_(xx) from the changein distance between the innermost finger FI and outermost finger FO withapproximately the following equation:

$\begin{matrix}{{{Hvs}\lbrack n\rbrack} = \frac{{d\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)} - {d\left( {{{fi}\left\lbrack {n - 1} \right\rbrack},{{FO}\left\lbrack {n - 1} \right\rbrack}} \right)}}{\Delta\; t}} & (66)\end{matrix}$where d(FI[n],FO[n]) is the squared Euclidean distance between thefingers:

$\begin{matrix}{d\left( {{{FI}\lbrack n\rbrack},{{{FO}\lbrack n\rbrack} = \sqrt{\left( {{{FI}_{x}\lbrack n\rbrack} - {{FO}_{x}\lbrack n\rbrack}} \right)^{2} + \left( {{{FI}_{y}\lbrack n\rbrack} - {{FO}_{y}\lbrack n\rbrack}} \right)^{2}}}} \right.} & (67)\end{matrix}$

If one of the innermost or outermost fingers was not touching during theprevious proximity image, the change in separation is assumed to bezero. Similarly, step 530 computes the hand rotational velocity H_(xx)from the change in angle between the innermost and outermost finger withapproximately the following equation:

$\begin{matrix}\begin{matrix}{{H_{vr}\lbrack n\rbrack} = {\left( \frac{{\angle\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right)} - {\angle\left( {{{FI}\left\lbrack {n - 1} \right\rbrack},{{FO}\left\lbrack {n - 1} \right\rbrack}} \right.}}{\Delta\; t} \right) \times}} \\{\left( \frac{d\left( {{{FI}\lbrack n\rbrack},{{FO}\lbrack n\rbrack}} \right.}{\pi} \right)}\end{matrix} & (68)\end{matrix}$

The change in angle is multiplied by the current separation to convertit to the same units as the translation and scaling components. Theseequations capture any rotation and scaling components of hand motioneven if the hand is also translating as a whole, thus making therotation and scaling degrees of freedom integral with translation.

Another reason the computations above are restricted to the thumb andpinky or innermost and outermost fingers is that users may want to makefine translating manipulations with the central fingers, i.e., index,middle, and ring, while the thumb and pinky remain stationary. Ifchanges in distances or angles between the central fingers and the thumbwere averaged with Equations 66-68, this would not be possible becausecentral finger translations would cause the appearance of rotation orscaling with respect to the stationary thumb or pinky. However,Equations 56-60 applied in the thumb verification process are onlysensitive to symmetric rotation and scaling about a fixed point betweenthe fingers. They approach zero if any significant whole handtranslation is occurring or the finger motions are not complementary. Incase the user fails to properly move the outermost finger during arotation or scaling gesture, step 531 uses equations of the approximateform of Equations 56-60 to compute rotation and scaling velocitiesbetween the thumb and any touching fingers other than the outermost. Theresulting velocities are preferably combined with the results ofEquations 66-68 via a maximum operation rather than an average in casetranslational motion causes the fixed point rotations or scalings to bezero. Finally, decision diamond 532 orders a check for radial orrotational deceleration 534 during motions prior to finger liftoff. Themethod for detecting radial or rotational deceleration is the same asthat detailed in the description of translation extraction.

FIG. 37 shows the details of hand translational velocity measurementsreferred to in process 506 of FIG. 34. The simplest way to compute ahand translation velocity would be to simply average the lateralvelocities of each finger. However, the user expects the motion orcontrol to display gain to be constant regardless of how many fingersare being moved, even if some are resting stationary. Furthermore, ifthe user is simultaneously scaling or rotating the hand, a simpleaverage is sensitive to spurious net translations caused: by uncanceledcentral finger motions.

Therefore, in a preferred embodiment the translational componentextractor carefully assigns weightings for each finger before computingthe average translation. Step 540 initializes the translation weightingFi_(vw) of each finger to its total contact proximity, i.e.,Fi_(vw)[n]=Fi_(z)[n]. This ensures that fingers not touching the surfacedo not dilute the average with their zero velocities and that fingerswhich only touch lightly have less influence since their position andvelocity measurements may be less reliable. The next step 544 decreasesthe weightings of fingers which are relatively stationary so that thecontrol to display gain of intentionally moving fingers is not diluted.This can be done by finding the fastest moving finger, recording itsspeed as a maximum finger speed and scaling each finger's translationweighting in proportion to its speed divided by the maximum of thefinger speeds, as shown approximately in the formula below:

$\begin{matrix}{{{Fi}_{vw}\lbrack n\rbrack}\text{:} = {{Fi}_{vw}\lbrack n\rbrack} \times \left( \frac{{Fi}_{speed}\lbrack n\rbrack}{\max_{j}{{Fi}_{speed}\lbrack n\rbrack}} \right)^{ptw}} & (69)\end{matrix}$where the power ptw adjusts the strength of the speed dependence. Notethat step 544 can be skipped for applications such ascomputer-aided-design in which users desire both a normal cursor motiongain mode and a low gain mode. Lower cursor motion gain is useful forfine, short range positioning, and would be accessed by moving only oneor two fingers while keeping the rest stationary.

Step 546 decreases the translation weightings for the central fingersduring hand scalings and rotations, though it does not prevent thecentral fingers from making fine translational manipulations while thethumb and pinky are stationary. The formulas below accomplish thisseamlessly by downscaling the central translation weightings as themagnitudes of the rotation and scaling velocities become significantcompared to K_(polathresh):

$\begin{matrix}{{{Fi}_{vwx}\lbrack n\rbrack} \approx \frac{{{Fi}_{vw}\lbrack n\rbrack} \times K_{polarthresh}}{K_{polarthresh} + {{H_{vr}\lbrack n\rbrack}}}} & (70) \\{{{Fi}_{vwy}\lbrack n\rbrack} \approx \frac{{{Fi}_{vw}\lbrack n\rbrack} \times K_{polarthresh}}{K_{polarthresh} + {{H_{vr}\lbrack n\rbrack}} + {{H_{vs}\lbrack n\rbrack}}}} & (71)\end{matrix}$where these equations are applied only to the central fingers whoseidentities i are between the innermost and outermost. Note that sincehand scaling does not cause much horizontal translation bias, thehorizontal translation weighting Fi_(vwx)[n] need not be affected byhand scaling velocity H_(vs)[n], as indicated by the lack of a handscaling term in Equation 70. The translation weightings of the innermostand outermost fingers are unchanged by the polar component speeds, i.e.,FI_(vwx)[n]=FI_(vwx)[n]=FI_(vw)[n] andFO_(vwx)[n]=FO_(vwx)=[n]=FO_(vw)[n]. Step 548 finally computes the handtranslation velocity vector (H_(vx)[n],H_(vy)[n]) from the weightedaverage of the finger velocities:

$\begin{matrix}{{H_{vx}\lbrack n\rbrack} = \frac{\sum\limits_{i - 1}^{5}{{Fi}_{vwx}{Fi}_{vx}}}{\sum\limits_{i - 1}^{5}{Fi}_{vwx}}} & (72) \\{{H_{vy}\lbrack n\rbrack} = \frac{\sum\limits_{i - 1}^{5}{{Fi}_{vwy}{Fi}_{vy}}}{\sum\limits_{i - 1}^{5}{Fi}_{vwy}}} & (73)\end{matrix}$

The last part of the translation calculations is to test for the lateraldeceleration of the fingers before liftoff, which reliably indicateswhether the user wishes cursor motion to stop at liftoff. Ifdeceleration is not detected prior to liftoff, the user may intendcursor motion to continue after liftoff, or the user may intend aspecial “one-shot” command to be invoked. Decision diamond 550 onlyinvokes the deceleration tests while finger proximities are not droppingtoo quickly, to prevent the perturbations in finger centroids which canaccompany finger liftoff from interfering with the decelerationmeasurements. Step 551 computes the percentage acceleration or ratio ofcurrent translation speed |H_(vx)[n],H_(vy)[n])| to a past averagetranslation speed preferably computed by a moving window average orautoregressive filter. Decision diamond 552 causes the translationdeceleration flag to be set 556 if the acceleration ratio is less than athreshold. If this threshold is set greater than one, the user will haveto be accelerating the fingers just prior to liftoff for cursor motionto continue. If the threshold is set just below one, cursor motion willreliably be continued as long as the user maintains a constant lateralspeed prior to liftoff, but if the user begins to slow the cursor onapproach to a target area of the display the deceleration flag will beset. Decision diamond 554 can also cause the deceleration flag to be setif the current translation direction is substantially different from anaverage of past directions. Such change in direction indicates the handmotion trajectory is curving, in which case cursor motion should not becontinued after liftoff because accurately determining the direction tothe user's intended target becomes very difficult. If neitherdeceleration nor curved trajectories are detected, step 558 clears thetranslation deceleration flag. This will enable cursor motioncontinuation should the fingers subsequently begin liftoff. Note thatdecision diamond 550 prevents the state of the translation decelerationflags from changing during liftoff so that the decision after liftoff tocontinue cursor motion depends on the state of the deceleration flagbefore liftoff began. The final step 560 updates the autoregressive ormoving window average of the hand translation velocity vector, which canbecome the velocity of continued cursor motion after liftoff. Actualgeneration of the continued cursor motion signals occurs in the chordmotion recognizer 18 as will be discussed with FIG. 40.

Note that this cursor motion continuation method has several advantagesover motion continuation methods in related art. Since the decision tocontinue motion depends on a percentage acceleration which inherentlynormalizes to any speed range, the user can intentionally invoke motioncontinuation from a wide range of speeds including very low speeds. Thusthe user can directly invoke slow motion continuation to auto scroll adocument at readable speeds. This is not true of Watanabe's method inU.S. Pat. No. 4,734,685, which only continues motion when the user'smotion exceeds a high speed threshold, nor of Logan et al.'s method inU.S. Pat. No. 5,327,161, which if enabled for low finger speeds willundesirably continue motion when a user decelerates on approach to alarge target but fails to stop completely before lifting off. Percentageacceleration also captures user intent more clearly than position of afinger in a border area. Position of a finger in a border area as usedin U.S. Pat. No. 5,543,591 to Gillespie et al. is ambiguous because thecursor can reach its desired target on the display just as the fingerenters the border, yet the touchpad device will continue cursor motionpast the target because it thinks the finger has run out of space tomove. In the present invention, on the other hand, the accelerationratio will remain near one if the fingers can slide off the edge of thesensing array without hitting a physical barrier, sensibly invokingmotion continuation. But if the fingers decelerate before crossing orstop on the edge of the sensing array, the cursor will stop as desired.

The details of the differential hand pressure extraction process 508 areshown in FIG. 38. Fingertip proximity, quickly saturates when pressureis applied through the bony tip normal to a hard surface. Unless thesurface itself is highly compliant, the best dynamic range of fingertippressure is obtained with the fingers outstretched and hand nearlyflattened so that the compressible soft pulp underneath the fingertipsrests on the surface. Decision diamond 562 therefore causes the tilt androll hand pressure components to be set to zero in step 564 and pressureextraction to abort unless the hand is nearly flattened. Inherent in thetest for hand flattening 562 is a finger count to ensure that most ofthe five fingers and both palm heels are touching the surface tomaximize the precision of the hand pressure measurements, thoughtechnically only three non-collinear hand contacts arranged like atripod are necessary to establish tilt and roll pressures. Decisiondiamond 562 can also require the user to explicitly enablethree-dimensional manipulation with an intuitive gesture such as placingall five fingers on the surface briefly tapping the palm heels on thesurface, and finally resting the palm heels on the surface. Decisiondiamond 566 causes step 568 to capture and store reference proximitiesfor each contact path when the proximity of all contacts have stabilizedat the end of this initiation sequence. The tilt and roll pressurecomponents are again zeroed 564 for the sensor array scan cycle duringwhich this calibration is performed.

However, during subsequent scan cycles the user can tilt the handforward applying more pressure to the fingertips or backward applyingmore pressure to the palm heels or the user can roll the hand outwardonto the pinky and outer palm heel or inward applying more pressure tothe thumb, index finger and inner palm heel. Step 5170 will proceed tocalculate an unweighted average of the current contact positions. Step572 computes for each hand part still touching the surface the ratio ofcurrent proximity to the reference proximity previously stored. To makethese ratios less sensitive to accidental lifting of hand parts, step574 clips them to be greater or equal to one so only increases inproximity and pressure register in, the tilt and roll measurements.Another average contact path position is computed in step 576, but thisone is weighted by the above computed proximity ratios for each path.The difference between these weighted and unweighted contact positionaverages taken in step 578 produces a vector whose direction canindicate the direction of roll or tilt and whose magnitude can controlthe rate of roll or tilt about x and y axes.

Since the weighted and unweighted position averages are only influencedby positions of currently contacting fingers and increases in contactpressure or proximity, the method is insensitive to finger liftoffs.Computation of reference-normalized proximity ratios in step 572 ratherthan absolute changes in proximity prevents the large palm heel contactsfrom having undue influence on the weighted average position.

Since only the current contact positions are used in the averageposition computations, the roll and tilt vector is independent oflateral motions such as hand translation or rotation as long as thelateral motions do not disturb finger pressure, thus once againachieving integrality. However, hand scaling and differential handpressure are difficult to use at the same time because flexing thefingers generally causes significant decreases in fingertip contact areaand thus interferes with inference of fingertip pressure changes. Whenthis becomes a serious problem, a total hand pressure component can beused as a sixth degree of freedom in place of the hand scalingcomponent. This total pressure component causes cursor velocity along az-axis in proportion to deviations of the average of the contactproximity ratios from one. Alternative embodiments may include furtherenhancements such as adapting the reference proximities to slowvariations in resting hand pressure and applying a dead zone filter toignore pressure difference vectors with small magnitudes

Despite the care taken to measure the polar velocity, translationvelocity, and hand pressure components in such a way that the resultantvectors are independent of one another, uneven finger motion during handscaling, rotation, or translation can still cause minor perturbations inmeasurements of one degree of freedom while primarily attempting to movein another. Non-linear filtering applied in steps 510 and 512 of FIG. 34removes the remaining motion leakage between dominant components andnearly stationary components. In steps 510 each component velocity isdownscaled by the ratio of its average speed to the maximum of all thecomponent speeds, the dominant component speed:

$\begin{matrix}{{H_{vx}\lbrack n\rbrack}\text{:} = {H_{vx}\lbrack n\rbrack} \times \left( \frac{H_{xyspeed}\lbrack n\rbrack}{dominantspeed} \right)^{pds}} & (74) \\{{{H_{vy}\lbrack n\rbrack}\text{:}} = {{H_{vy}\lbrack n\rbrack} \times \left( \frac{H_{xyspeed}\lbrack n\rbrack}{dominantspeed} \right)^{pds}}} & (75) \\{{{H_{vs}\lbrack n\rbrack}\text{:}} = {{H_{vs}\lbrack n\rbrack} \times \left( \frac{H_{sspeed}\lbrack n\rbrack}{dominantspeed} \right)^{pds}}} & (76) \\{{{H_{vr}\lbrack n\rbrack}\text{:}} = {{H_{vr}\lbrack n\rbrack} \times \left( \frac{H_{rspeed}\lbrack n\rbrack}{dominantspeed} \right)^{pds}}} & (77)\end{matrix}$where H_(xyspeed)[n], H_(sspeed)[n], and H_(rspeed)[n] areautoregressive averages over time of the translation speed, scalingspeed, and rotational speed, where:dominant_speed=max(H _(xyspeed) [n],H _(sspeed) [n],H _(rspeed)[n])  (78)where pds controls the strength of the filter. As pdy is adjustedtowards infinity the dominant component is picked out and all componentsless than the dominant tend toward zero producing the orthogonal cursoreffect well-known in drawing applications. As pds is adjusted towardszero the filters have no effect. Preferably, pds is set in between sothat components significantly slower than the dominant are slowedfurther, but components close to the dominant in speed are barelyaffected, preserving the possibility of diagonal motion in multipledegrees of freedom at once. The autoregressive averaging helps to pickout the component or components which are dominant over the long termand suppress the others even while the dominant components are slowingto a stop.

Step 512 takes a second pass with a related filter known as a dead-zonefilter. A dead-zone filter produces zero output velocity for inputvelocities less than a speed threshold but produces output speeds inproportion to the difference between the input speed and the thresholdfor input velocities that exceed the threshold. Preferably the speedthreshold or width of the dead zone is set to a fraction of the maximumof current component speeds. All velocity components are filtered usingthis same dead zone width. The final extracted component velocities areforwarded to the chord motion recognizer module 18 which will determinewhat if any input events should be generated from the motions.

FIG. 39A shows the details of the finger synchronization detector module14. The synchronization detection process described below is repeatedfor each hand independently. Step 600 fetches proximity markers andidentifications for the hand's current paths. The identifications willbe necessary to ignore palm paths and identify combinations ofsynchronized fingers, while the proximity markers record the time atwhich each contact path first exceeds a press proximity threshold andthe time at which each contact path drops below a release proximitythreshold prior to total liftoff. Setting these proximity thresholdssomewhat higher than the minimum proximity considered significant by thesegmentation search process 264, produces more precise finger press andrelease times.

Step 603 searches for subsets of fingers which touch down at about thesame time and for subsets of fingers which lift off at about the sametime. This can be done by recording each finger path along with itspress time in a temporally ordered list as it crosses the pressproximity threshold. Since the primary function of the palms is tosupport the forearms while the hands are resting, palm activity isignored by the typing 12 and chord motion recognizers 18 except duringdifferential hand pressure extraction and palm heel presses can beexcluded from this list and most other synchronization tests. To checkfor synchronization between the two most recent finger presses, thepress times of the two most recent entries in the list are compared. Ifthe difference between their press times is less than a temporalthreshold, the two finger presses are considered synchronized. If not,the most recent finger press is considered asynchronous. Synchronizationamong three or more fingers up to five is found by comparing press timesof the three, four, or five most recent list entries. If the press timeof the most recent entry is within a temporal threshold of the nth mostrecent entry, synchronization among the n most recent finger presses isindicated. To accommodate imprecision in touchdown across the hand, themagnitude of the temporal threshold should increase slightly inproportion to the number of fingers being tested for synchronization.The largest set of recent finger presses found to be synchronized isrecorded as the synchronized subset, and the combination of fingeridentities comprising this subset is stored conveniently as a fingeridentity bitfield. The term subset is used because the synchronizedpress subset may not include all fingers currently touching the surface,as happens when a finger touches down much earlier than the otherfingers yet remains touching as they simultaneously touch down. Anordered list of finger release times is similarly maintained andsearched separately. Alternative embodiments may require that a fingerstill be touching the surface to be included in the synchronized presssubset.

Decision diamond 602 checks whether a synchronization marker is pendingfrom a previous image scan cycle. If not, decision diamond 604 checkswhether the search 603 found a newly synchronized press subset in thecurrent proximity image. If so, step 606 sets the temporalsynchronization marker to the oldest press within the new synchronizedsubset. Additional finger presses may be added to the subset duringfuture scan cycles without affecting the value of this temporalsynchronization marker. If there is currently no finger presssynchronization, decision diamond 605 determines whether three or morefingers have just been released simultaneously. Simultaneous release ofthree or more fingers should not occur while typing with a set offingers but does occur when lifting fingers off the surface from rest.Therefore simultaneous release of three or more fingers reliablyindicates that the released fingers are not intended as keypresses andshould be deleted from the keypress queue 605, regardless of whetherthese same fingers touched down synchronously. Release synchronizationof two fingers is not by itself a reliable indicator of typing intentand has no effect on the keypress queue. The keypress queue is describedlater with FIGS. 42-43B.

Once a press synchronization marker for the hand is pending, furtherprocessing checks the number of finger presses which are synchronizedand waits for release of the synchronized fingers. If decision diamond608 finds three or more fingers in the synchronized press subset theuser cannot possibly be typing with these fingers. Therefore step 612immediately deletes the three or more synchronized presses from thekeypress queue. This way they cannot cause key symbol transmission tothe host, and transmission of key symbols from subsequent asynchronouspresses is not blocked waiting for the synchronized fingers to bereleased.

However, when the synchronization only involves two finger presses 608,it is difficult to know whether the user intended to tap a finger pairchord or intended to type two adjacent keys and accidentally let the keypresses occur simultaneously. Since such accidental simultaneous pressesare usually followed by asynchronous releases of the two fingers, butfinger pair chords are usually released synchronously, the decisionwhether the presses are asynchronous key taps or chord taps must bedelayed until finger release can be checked for synchronization. In themeantime, step 610 places a hold on the keypress queue to preventtransmission of key symbols from the possible finger chord or anysubsequent finger presses. To prevent long backups in key transmission,decision diamond 614 will eventually release the queue hold by havingstep 615 delete the synchronized presses from the keypress queue if bothfingers remain touching a long time. Though this aborts the hypothesisthat the presses were intended as key taps, the presses are also lesslikely to be key taps if the fingers are not lifted soon aftertouchdown.

If the synchronized fingers are not lifting, decision diamond 616 leavesthe synchronization marker pending so synchronization checks can becontinued with updated path parameters 600 after the next scan cycle. Ifthe synchronized fingers are lifting, but decision diamond 618 findswith the help of the synchronization release search 603 that they aredoing so asynchronously 618, step 622 releases any holds on the keypressqueue assuming any synchronized finger pair was intended to be twokeypresses. Though the synchronized finger presses are not deleted fromthe keypress queue at this point, they may have already been deleted instep 612 if the pressed subset contained more than two. Also, step 624clears the temporal synchronization marker, indicating that no furthersynchronization tests need be done for this subset.

Continuing to FIG. 39B, if the fingers synchronized during touchdownalso lift simultaneously, step 618 removes them and any holds from thekeypress queue in case they were a pair awaiting a positive releasesynchronization test. Further tests ensue to determine whether thesynchronized fingers meet additional chord tap conditions. As withsingle finger taps, the synchronized fingers cannot be held on thesurface more than about half a second if they are to qualify, as a chordtap. Decision diamond 626 tests this by thresholding the time betweenthe release of the last remaining synchronized finger and the temporalpress synchronization marker. A chord tap should also exhibit a limitedamount of lateral finger motion, measured either as an average of peakfinger speeds or distance traveled since touchdown in decision diamond628. If the quick release and limited lateral motion conditions are notmet, step 624 clears the synchronization marker with the conclusion thatthe synchronized fingers were either just resting fingers or part of achord slide.

If the chord tap conditions are met, step 630 looks up, using thesynchronized subset bitfield, any input events such as mouse clicks orkeyboard commands assigned to the combination of fingers in the chordtap. Some chords such as those including all four fingertips may bereserved as resting chords 634, in which case decision diamond 632 willfind they have no associated input events. If the chord does have tapinput events, step 636 appends these to the main outgoing event queue ofthe host communication interface 20. Finally step 624 clears thesynchronization marker in readiness for future finger synchronizationson the given hand.

As a further precaution against accidental generation of chord tapswhile typing, it is also useful for decision diamond 632 to ignorethrough step 634 the first chord tap which comes soon after a validkeypress without a chord slide in between. Usually after typing the userwill need to reposition the mouse cursor before clicking, requiring anintervening chord slide. If the mouse cursor happens to already be inplace after typing, the user may have to tap the finger chord a secondtime for the click to be sent, but this is less risky than having anaccidental chord tap cause an unintended mouse button click in themiddle of a typing session.

FIG. 40A shows the detailed steps of the chord motion recognizer module18. The chord motion recognition process described below is repeated foreach hand independently. Step 650 retrieves the parameters of the hand'sidentified paths 250 and the hand's extracted motion components from themotion extraction module 16. If a slide of a finger chord has notalready started, decision diamond 652 orders slide initiation tests 654and 656. To distinguish slides from glancing finger taps during typing,decision diamond 654 requires at least two fingers from a hand to betouching the surface for slide mode to start. There may be someexceptions to this rule, such as allowing a single finger to resume aprevious slide within a second or so after the previous slide chordlifts off the surface.

In a preferred embodiment, the user can start a slide and specify itschord in either of two ways. In the first way, the user starts with thehand floating above the surface, places some fingers on the surfacepossibly asynchronously, and begins moving all of these fingerslaterally. Decision diamond 656 initiates the slide mode only whensignificant motion is detected in all the touching fingers. Step 658selects the chord from the combination of fingers touching whensignificant motion is detected, regardless of touchdown synchronization.In this case coherent initiation of motion in all the touching fingersis sufficient to distinguish the slide from resting fingers, sosynchronization of touchdown is not necessary. Also, novice users mayerroneously try to start a slide by placing and sliding only one fingeron the surface, forgetting that multiple fingers are necessary.Tolerance of asynchronous touchdown allows them to seamlessly correctthis by subsequently placing and sliding the rest of the fingers desiredfor the chord. The slide chord will then initiate without forcing theuser to pick up all fingers and start over with synchronized fingertouchdowns.

In the second way, the user starts with multiple fingers resting on thesurface, lifts a subset of these fingers, touches a subset back down onthe surface synchronously to select the chord, and begins moving thesubset laterally to initiate the slide. Decision diamond 656 actuallyinitiates the slide mode when it detects significant motion in all thefingers of the synchronized subset. Whether the fingers which remainedresting on the surface during this sequence begin to move does notmatter since in this case the selected chord is determined in step 658by the combination of fingers in the synchronized press subset, not fromthe set of all touching fingers. This second way has the advantage thatthe user does not have to lift the whole hand from the surface beforestarting the slide, but can instead leave most of the weight of thehands resting on the surface and only lift and press the two or threefingers necessary to identify the most common finger chords.

To provide greater tolerance for accidental shifts in resting fingerpositions, decision diamond 656 requires both that all relevant fingersare moving at significant speed and that they are moving about the samespeed. This is checked either by thresholding the geometric mean of thefinger speeds or by thresholding the fastest finger's speed andverifying that the slowest finger's speed is at least a minimum fractionof the fastest finger's speed. Once a chord slide is initiated, step 660disables recognition of key or chord taps by the hand at least untileither the touching fingers or the synced subset lifts off.

Once the slide initiates, the chord motion recognizer could simply beginsending raw component velocities paired with the selected combination offinger identities to the host. However, in the interest of backwardcompatibility with the mouse and key event formats of conventional inputdevices, the motion event generation steps in FIG. 40B convert motion inany of the extracted degrees of freedom into standard mouse and keycommand events which depend on the identity of the selected chord. Tosupport such motion conversion, step 658 finds a chord activitystructure in a lookup table using a bitfield of the identities of eitherthe touching fingers or the fingers in the synchronized, subset.Different finger identity combinations can refer to the same chordactivity structure. In the preferred embodiment, all finger combinationswith the same number of non-thumb fingertips refer to the same chordactivity structure, so slide chord activities are distinguished bywhether the thumb is touching and how many non-thumb fingers aretouching. Basing chord action on the number of fingertips rather thantheir combination still provides up to seven chords per hand yet makeschords easier for the user to memorize and perform. The user has thefreedom to choose and vary which fingertips are used in chords requiringonly one; two or three fingertips. Given this freedom, users naturallytend to pick combinations in which all touching fingertips are adjacentrather than combinations in which a finger such as the ring finger islifted but the surrounding fingers such as the middle and pinky musttouch. One chord typing study found that users can tap these fingerchords in which all pressed fingertips are adjacent twice as fast asother chords.

The events in each chord activity structure are organized into slices.Each slice contains events to be generated in response to motion in aparticular range of speeds and directions within the extracted degreesof freedom. For example, a mouse cursor slice could be allocated anytranslational speed and direction. However, text cursor manipulationrequires four slices, one for each arrow key, and each arrow's sliceintegrates motion in a narrow direction range of translation. Each slicecan also include motion sensitivity and so-called cursor accelerationparameters for each degree of freedom. These will be used to discretizemotion into the units such as arrow key clicks or mouse clicks expectedby existing host computer systems.

Step 675 of chord motion conversion simply picks the first slice in thegiven chord activity structure for processing. Step 676 scales thecurrent values of the extracted velocity components by the slice'smotion sensitivity and acceleration parameters. Step 677 geometricallyprojects or clips the scaled velocity components into the slice'sdefined speed and direction range. For the example mouse cursor slice,this might only involve clipping the rotation and scaling components tozero. But for an arrow key slice, the translation velocity vector isprojected onto the unit vector pointing in the same direction as thearrow. Step 678 integrates each scaled and projected component velocityover time in the slice's accumulators until decision diamond 680determines at least one unit of motion has been accumulated. Step 682looks up the slice's preferred mouse, key, or three-dimensional inputevent format, attaches the number of accumulated motion units to theevent; and step 684 dispatches the event to the outgoing queue of thehost communication interface 20. Step 686 subtracts the sent motionevents from the accumulators, and step 688 optionally clears theaccumulators of other slices. If the slice is intended to generate asingle key command per hand motion, decision diamond 689 will determinethat it is a one-shot slice so that step 690 can disable further eventgeneration from it until a slice with a different direction intervenes.If the given slice is the last slice, decision diamond 692 returns tostep 650 to await the next scan of the sensor array. Otherwise step 694continues to integrate and convert the current motion for other slices.

Returning to FIG. 40A, for some applications it may be desirable tochange the selected chord whenever an additional finger touches down orone of the fingers in the chord lifts off. However, in the preferredembodiment, the selected chord cannot be changed after slide initiationby asynchronous finger touch activity. This gives the user freedom torest or lift addition fingers as may be necessary to get the bestprecision in a desired degree of freedom. For example, even though thefinger pair chord does not include the thumb, the thumb can be set downshortly after slide initiation to access the full dynamic range of therotation and scaling degrees of freedom. In fact, all remaining liftedfingers can always be set down after initiation of any chord to allowmanipulation by the whole hand. Likewise, all fingers but one can belifted, yet translation will continue.

Though asynchronous finger touch activity is ignored, synchronizedlifting and pressing of multiple fingers subsequent to slide initiationcan create a new synchronized subset and change the selected chord.Preferably this is only allowed while the hand has paused but itsfingers are still resting on the surface. Decision diamond 670 willdetect the new subset and commence motion testing in decision diamond673 which is analogous to decision diamond 656. If significant motion isfound in all fingers of the newly synchronized subset, step 674 willselect the new subset as the slide chord and lookup a new chord activitystructure in analogy to step 658. Thus finger synchronization againallows the user to switch to a different activity without forcing theuser to lift the whole hand from the surface. Integration of velocitycomponents resumes but the events generated from the new chord activitystructure will presumably be different.

It is advantageous to provide visual or auditory feedback to the userabout which chord activity structure has been selected. This can beaccomplished visually by placing a row of five light emitting diodesacross the top of the multi-touch surface, with one row per hand to beused on the surface. When entering slide mode, step 658 would turn on acombination of these lights corresponding to the combination of fingersin the selected chord. Step 674 would change the combination of activelights to match the new chord activity structure should the user selecta new, chord, and step 668 would turn them off. Similar lights could beemulated on the host computer display 24. The lights could also beflashed to indicate the finger combination detected during chord taps instep 636. The implementation for auditory feedback would be similar,except light combinations would be replaced with tone or tone burstcombinations.

The accumulation and event generation process repeats for all array scancycles until decision diamond 664 detects liftoff by all the fingersfrom the initiating combination. Decision diamond 666 then checks thepre-liftoff deceleration flag of the dominant motion, component. Thestate of this flag is determined by step 556 or 558 of translationextraction (FIG. 37) if translation is dominant, or by correspondingflags in step 534 of polar extraction. If there has been significantdeceleration, step 668 simply exits the chord slide mode, setting theselected chord to null. If the flag indicates no significant fingerdeceleration prior to liftoff, decision diamond 666 enables motioncontinuation mode for the selected chord. While in this mode, step 667applies the pre-liftoff weighted average (560) of dominant componentvelocity to the motion accumulators (678) in place of the currentvelocities, which are presumably zero since no fingers touch thesurface. Motion continuation mode does not stop until any of theremaining fingers not in the synchronized subset are lifted or morefingers newly touch down. This causes decision diamond 664 to becomefalse and normal slide activity with the currently selected chord toresume. Though the cursor or scrolling velocity does not decay duringmotion continuation mode, the host computer can send a signalinstructing motion continuation mode to be canceled if the cursorreaches the edge of the screen or end of a document. Similarly, if anyfingers remain on the surface during motion continuation, theirtranslations can adjust the cursor or scrolling velocity.

In the preferred embodiment, the chord motion recognizers for each handfunction independently and the input events for each chord can beconfigured independently. This allows the system to allocate tasksbetween hands in many different ways and to support a variety ofbimanual manipulations. For example, mouse cursor motion can beallocated to the fingertip pair chord on both hands and mouse buttondrag to a triple fingertip chord on both hands. This way the mousepointer can be moved and drug with either hand on either half of thesurface. Primary mouse clicks would be generated by a tap of a fingertippair on either half of the surface, and double-clicks could beergonomically generated by a single tap of three fingertips on thesurface. Window scrolling could be allocated to slides of four fingerson either hand.

Alternatively, mouse cursor manipulations could be allocated asdiscussed above to the right hand and right half of the surface, whilecorresponding text cursor manipulations are allocated to chords on theleft hand. For instance, left fingertip pair movement would generatearrow key commands corresponding to the direction of motion, and threefingertips would generate shift arrow combinations for selection oftext.

For host computer systems supporting manipulations in three or moredegrees of freedom, a left hand chord could be selected to pan, zoom,and rotate the display background while a corresponding chord in theright hand could translate, resize and rotate a foreground object. Thesechords would not have to include the thumb since the thumb can touchdown anytime after initiating chord motion without changing the selectedchord. The user then need add the thumb to the surface when attemptingrotation or scaling.

Finger chords which initially include the thumb can be reserved forone-shot command gestures, which only generate input events once foreach slide of a chord rather than repeating transmission each time anadditional unit of motion is detected. For example, the common editingcommands cut, copy and paste can be intuitively allocated to a pinchhand scaling, chord tap, and anti-pinch hand scaling of the thumb and anopposing fingertip.

FIG. 41 shows the steps within the key layout definition and morphingprocess, which is part of the typing recognition module 12. Step 700retrieves at system startup a key layout which has been pre-specified bythe user or manufacturer. The key layout consists of a set of key regiondata structures. Each region has associated with it the symbol orcommands which should be sent to the host computer when the region ispressed and coordinates representing the location of the center of theregion on the surface. In the preferred embodiment, arrangement of thosekey regions containing alphanumeric and punctuation symbols roughlycorresponds to either the QWERTY or the Dvorak key layouts common onmechanical keyboards.

In some embodiments of the multi-touch surface apparatus it isadvantageous to be able to snap or morph the key layout to the restingpositions of the hands. This is especially helpful for multi-touchsurfaces which are several times larger than the standard keyboard orkey layout, such as one covering an entire desk. Fixing the key layoutin one small fixed area of such a surface would be inconvenient anddiscourage use of the whole available surface area. To provide feedbackto the user about changes in the position of the key layout, theposition of the key symbols in these embodiments of the multi-touchsurface would not be printed permanently on the surface. Instead, theposition of the key symbols would be reprogrammably displayed on thesurface by light emitting polymers, liquid crystal, or other dynamicvisual display means embedded in the multi-touch surface apparatus alongwith the proximity sensor arrays.

Given such an apparatus, step 702 retrieves the current paths from bothhands and awaits what will be known as a layout homing gesture. Ifdecision diamond 704 decides with the help of, a hand's synchronizationdetector that all five of the hand's fingers have just been placed onthe surface synchronously, step 706 will attempt to snap the key layoutto the hand such that the hand's home row keys lie under thesynchronized fingertips, wherever the hand is on the surface. Step 706retrieves the measured hand offsets from the hand position estimator andtranslates all key regions which are normally typed by the given hand inproportion to the measured hand offsets. Note the currently measuredrather than filtered estimates of offsets can be used because when allfive fingers are down there is no danger of finger misidentificationcorrupting the measured offsets. This procedure assumes that theuntranslated locations of the home row keys are the same as the defaultfinger locations for the hand.

Decision diamond 708 checks whether the fingers appear to be in aneutral, partially closed posture, rather closed than outstretched orpinched together. If the posture is close to neutral, step 710 mayfurther offset the keys normally typed by each finger, which for themost part are the keys in the same column of the finger by the measuredfinger offsets. Temporal filtering of these finger offsets over severallayout homing gestures will tend to scale the spacing between columns ofkeys to the user's hand size. Spacing between rows is scaled down inproportion to the scaling between columns.

With the key layout for the hand's keys morphed to fit the size andcurrent position of the resting hand, step 712 updates the displayedposition of the symbols on the surface, so that the user will see thatthe key layout has snapped to the position of his hand. From this stagethe user can begin to type and the typing recognizer 718 will use themorphed key region locations to decide what key regions are beingpressed. The layout will remain morphed this way until either the userperforms another homing gesture to move it somewhere else on thesurface, or until the user takes both hands off the surface for a while.Decision diamond 714 will eventually time out so that step 716 can resetthe layout to its default position in readiness for another user orusage session.

For smaller multi-touch surfaces in which the key layout is permanentlyprinted on the surface, it is advantageous to give the user tactilefeedback about the positions of key regions. However, any tactileindicators placed on the surface must be carefully designed so as not toimpede smooth sliding across the surface. For example, shallowdepressions made in the surface near the center of each key mimickingthe shallow depressions common on mechanical keyboard keycaps wouldcause a vibratory washboard effect as the hand slides across thesurface. To minimize such washboard effects, in the preferred embodimentthe multi-touch surface provides for the fingertips of each hand asingle, continuous depression running from the default index fingertiplocation to the default pinky fingertip location. This corresponds onthe QWERTY key layout to shallow, slightly arched channels along homerow from the “J” key to the “;” key for the right hand, and from the “A”key to the “F” key for the left hand. Similarly, the thumbs can each beprovided with a single oval-shaped depression at their defaultlocations, slanted slightly from vertical to match the default thumborientation. These would preferably correspond to “Space” and“BackSpace” key regions for the right and left thumbs, respectively.Such minimal depressions can tactilely guide users' hands back to homerow of the key layout without requiring users to look down at thesurface and without seriously disrupting finger chord slides andmanipulations on the surface.

The positions of key regions off home row can be marked by other typesof tactile indicators. Simply roughening the surface at key regions doesnot work well. Though humans easily differentiate textures when slidingfingers over them, most textures cannot be noticed during quick taps ona textured region. Only relatively abrupt edges or protrusions can besensed by the users' fingertips under typing conditions. Therefore, asmall raised dot like a Braille dot is formed on top of the surface atthe center of each key region. The user receives feedback on theaccuracy of their typing strokes from where on the fingertip a dot isfelt. This feedback can be used to correct finger aim during futurekeypresses. Since single finger slides are ignored by the chord motionrecognizer, the user can also slide a finger around the surface intactile search of a particular key region's dot and then tap the keyregion when the dot is found, all without looking at the surface. Eachdot should be just: large enough to be felt during tapping but not solarge as to impede chord slides across the surface. Even if the dots arenot large enough to impede sliding, they can still corrupt proximity andfingertip centroid measurements by raising the fingertip flesh near thedot off the surface thus locally separating the flesh from theunderlying proximity sensing electrode. Therefore, in the preferredembodiment, the portion of each dot above the surface dielectric is madeof a conductive material. This improves capacitive coupling between theraised fingertip flesh and the underlying electrodes.

FIG. 42 shows the steps within the keypress detection loop. Step 750retrieves from the current identified path data 250 any paths which wererecently created due to hand part touchdown or the surface. Decisiondiamond 752 checks whether the path proximity reached a keypressproximity thresh for the first time during the current sensor arrayscan. If the proximity has not reached the threshold yet or has alreadyexceeded it previously, control returns to step 750 to try keypressdetection on the next recent path. If the path just crossed the keypressproximity threshold decision diamond 754 checks whether the contact pathhas been identified as a finger rather than a palm. To give the usersthe freedom rest the palms anywhere on the surface, palm presses shouldnot normally cause keypresses, and are therefore ignored. Assuming thepath is a finger, decision diamond 756 checks whether the hand theidentified finger comes from is currently performing a chord slidegesture or writing via the pen grip hand configuration. Asynchronousfinger presses are ignored once these activities have started, as alsoindicated in step 660 of FIG. 40A. Assuming such hand activities are notongoing, decision diamond 757 proceeds with debounce tests which checkthat the finger has touched the surface for at least two sensor arrayscan cycles and that it had been off the surface for several scan cyclesbefore touching down. The path tracking module (FIG. 22) facilitatessuch liftoff debouncing by reactivating in step 334 a finger's old pathif the finger lifts off and quickly touches back down over the samespot. Upon reactivation the time stamp of the last liftoff by the oldpath must be preserved for comparison with the time stamp of the newtouchdown.

If all of these tests are passed, step 758 looks up the current pathposition (P_(x)[n],P_(y)[n]), and step 760 finds the key region whosereference position is closest to the fingertip centroid. Decisiondiamond 762 checks that the nearest region is within a reasonabledistance of the finger, and if not causes the finger press to beignored. Assuming a key region is close to the finger, step 764 createsa keypress element data structure containing the path, index identifierand finger identity, the closest key region, and a time stamp indicatingwhen the finger crossed the keypress proximity threshold. Step 766 thenappends this element data structure to the tail of a FIFO keypressqueue. This accomplished, processing returns to step 750 to process orwait for touchdowns by other fingers.

The keypress queue effectively orders finger touchdowns by when theypass the keypress transmitted to the host. However, an element's keysymbol is not assured transmission of the host once in the keypressqueue. Any of a number of conditions such as being part of asynchronized subset of pressing fingers can cause it to be deleted fromthe queue before being transmitted to the host. In this sense thekeypress queue should be considered a keypress candidate queue. Unlikethe ordered lists of finger touchdowns and releases maintained for eachhand separately in the synchronization detector, the keypress queueincludes and orders the finger touchdowns from both hands.

FIG. 43A shows the steps within the keypress acceptance and transmissionloop. Step 770 picks the element at the head of the keypress queue,which represents the oldest finger touchdown which has neither beendeleted from the queue as an invalid keypress candidate nor transmittedits associated key symbol. Decision diamond 772 checks whether the pathis still identified as a finger. While waiting in the queue pathproximity could have increased so much that the identification systemdecides the path is actually from a palm heel, in which case step 778deletes the keypress element without transmitting to the host and step770 advances processing to the next element. Decision diamond 774 alsoinvalidates the element if its press happened synchronously with otherfingers of the same hand. Thus decision diamond 774 follows through ondeletion command steps 601, 612, 615, 620 of the synchronizationdetection process (FIG. 39). Decision diamond 776 invalidates thekeypress if too much lateral finger motion has occurred since touchdown,even if that lateral finger motion has not yet caused a chord slide tostart. Because users may be touch typing on the surface, severalmillimeters of lateral motion are allowed to accommodate glancingfingertip motions which often occur when quickly reaching for keys. Thisis much more glancing tap motion than is tolerated by touchpads whichemploy a single finger slide for mouse cursor manipulation and a singlefinger tap for key or mouse button click emulation.

Decision diamond 780 checks whether the finger whose touchdown createdthe keypress element has since lifted off the surface. If so, decisiondiamond 782 checks whether it was lifted off soon enough to qualify as anormal key tap. If so, step 784 transmits the associated key symbol tothe host and step 778 deletes it from the head of the queue. Note that akeypress is always deleted from the queue upon liftoff, but even thoughit may have stayed on the surface for a time exceeding the tap timeout,it may have still caused transmission as a modifier key, as an impulsivepress with hand resting, or as a typematic press, as described below.

When a keypress is transmitted to the host it is advantageous for asound generation device on the multi-touch surface apparatus or hostcomputer to emit an audible click or beep as feedback to the user.Generation of audible click and beep feedback in response to keypressesis well known in commercial touchscreens, kiosks, appliance controlpanels and mechanical keyboards in which the keyswitch action is nearlysilent and does not have a make force threshold which feels distinctiveto the user. Feedback can also be provided as a light on the multi-touchsurface apparatus which flashes each time a keypress is sent. Keypressesaccompanied by modifier keypresses should cause longer flashes or tonesto acknowledge that the key symbol includes modifiers.

If the finger has not yet lifted, decision diamond 786 checks whetherits associated key region is a modifier such as <shift>, <ctrl>, or<alt>. If so, step 788 advances to the next element in the queue withoutdeleting the head. Processing will continue at step 772 to see if thenext element is a valid key tap. If the next element successfullyreaches the transmission stage, step 784 will scan back toward the headof the queue for any modifier regions which are still pressed. Then step784 can send the next element's key symbol along with the modifyingsymbols of any preceding modifier regions.

Decision diamond 782 requires that users touch the finger on the surfaceand lift back off within a few hundred milliseconds for a key to besent. This liftoff timing requirement substitutes for the forceactivation threshold of mechanical keyswitches. Like the force thresholdof mechanical keyswitches, the timing constraint provides a way for theuser to rest the finger on the key surface without invoking a keypress.The synchronization detector 14 provides another way forefingers to reston the surface without generating key symbols: they must touch down atthe same time as at least one other finger. However, sometimes userswill start resting by simultaneously placing the central fingertips onthe surface, but then they follow asynchronously with the pinky a secondlater and the thumb a second after that. These latter presses areessentially asynchronous and will not be invalidated by thesynchronization detector, but as long as they are not lifted within acouple hundred milliseconds, decision diamond 782 will delete themwithout transmission. But, while decision diamond 782 provides toleranceof asynchronous finger resting, its requirement that fingers quicklylift off, i.e., crisply tap, the surface to cause key generation makesit very difficult to keep most of the fingers resting on the surface tosupport the hands while tapping long sequences of symbols. This causesusers to raise their hands off the surface and float them above thesurface during fast typing sequences. This is acceptable typing postureexcept that the users arms will eventually tire if the user fails torest the hands back on the surface between sequences.

To provide an alternative typing posture which does not encouragesuspension of the hands above the surface, decision diamond 790 enablesa second key acceptance mode which does not require quick finger liftoffafter each press. Instead, the user must start with all five fingers ofa hand resting on the surface. Then each time a finger is asynchronouslyraised off the surface and pressed on a key region, that key region willbe transmitted regardless of subsequent liftoff timing. If the surfaceis hard such that fingertip proximity quickly saturates as force isapplied, decision diamond 792 checks the impulsivity of the proximityprofile for how quickly the finger proximity peaks. If the proximityprofile increases to its peak very slowly over time, no key will begenerated. This allows the user to gently set down a raised fingerwithout generating a key in case the user lifts the finger with theintention of generating a key but then changes his mind. If the touchsurface is compressible, decision diamond 792 can more directly inferfinger force from the ratio of measured fingertip proximity to ellipseaxis lengths. Then it can threshold the inferred force to distinguishdeliberate key presses from gentle finger rests. Since when intending togenerate a key the user will normally press down on the new key regionquickly after lifting off the old key region, the impulsivity and forcethresholds should increase with the time since the finger lifted off thesurface.

Emulating typematic on a multi-touch surface presents special problemsif finger resting force cannot be distinguished reliably from sustainedholding force on a key region. In this case, the special touch timingsequence detected by the steps of FIG. 43B supports reliable typematicemulation. Assuming decision diamond 798 finds that typematic has notstarted yet, decision diamond 794 checks whether the keypress queueelement being processed represents the most recent finger touchdown onthe surface. If any finger touchdowns have followed the touchdownrepresented by this element, typematic can never start from this queueelement. Instead, decision diamond 796 checks whether the element'sfinger has been touching longer than the normal tap timeout. If thefinger has been touching too long, step 778 should delete its keypresselement because decision diamond 786 has determined it is not a modifierand decision diamond 794 has determined it can never start typematic. Ifdecision diamond 794 determines that the keypress element does notrepresent the most recent touchdown, yet decision diamond 796 indicatesthe element has not exceeded the tap timeout, processing returns to step770 to await either liftoff or timeout in a future sensor array scan.This allows finger taps to overlap in the sense that a new key regioncan be pressed by a finger before another finger lifts off the previouskey region. However, either the press times or release times of such apair of overlapping finger taps must be asynchronous to prevent the pairfrom being considered a chord tap.

Assuming the finger touchdown is the most recent, decision diamond 800checks whether the finger has been touching for a typematic hold setupinterval of between about half a second and a second. If not, processingreturns to 770 to await either finger liftoff or the hold setupcondition to be met during future scans of the sensor array. When thehold setup condition is met, decision diamond 802 checks whether allother fingers on the hand of the given finger keypress lifted off thesurface more than a half second ago. If they did, step 804 willinitialize typematic for the given keypress element. The combination ofdecision diamonds 800 and 802 allow the user to have other fingers ofthe hand to be resting on the surface when a finger intended fortypematic touches down. But typematic will not start unless the otherfingers lift off the surface within half a second of the desiredtypematic finger's touchdown, and typematic will also not start untilthe typematic finger has a continued to touch the surface for at leasthalf a second after the others lifted off the surface. If thesestringent conditions are not met, the keypress element will not starttypematic and will eventually be deleted through either tap timeout 782when the finger lifts off or through tap timeout 796) if another touchesdown after it.

Step 804 simply sets a flag which will indicate to decision diamond 798during future scan cycles that typematic has already started for theelement. Upon typematic initialization, step 810 sends out the keysymbol for the first time to the host interface communication queue,along with any modifier symbols being held down by the opposite hand.Step 812 records the time the key symbol is sent for future reference bydecision diamond 808. Processing then returns to step 770 to await thenext proximity image scan.

Until the finger lifts off or another taps asynchronously, processingwill pass through decision diamond 798 to check whether the key symbolshould be sent again. Step 806 computes the symbol repeat intervaldynamically to be inversely proportional to finger proximity. Thus thekey will repeat faster as the finger is pressed on the surface harder ora larger part of the fingertip touches the surface. This also reducesthe chance that the user will cause more repeats than intended since asfinger proximity begins to drop during liftoff the repeat intervalbecomes much longer. Decision diamond 808 checks whether the dynamicrepeat interval since the last typematic symbol send has elapsed, and ifnecessary sends the symbol again in 810 and updates the typematic sendtime stamp 812.

It is desirable to let the users rest the other fingers back onto thesurface after typematic has initiated 804 and while typematic continues,but the user must do so without tapping. Decision diamond 805 causestypematic to be canceled and the typematic element deleted 778 if theuser asynchronously taps another finger on the surface as if trying tohit another key. If this does not occur, decision diamond 182 willeventually cause deletion of the typematic element when its finger liftsoff.

The typing recognition process described above thus allows themulti-touch surface to ergonomically emulate both the typing and handresting capabilities of a standard mechanical keyboard. Crisp taps orimpulsive presses on the surface generate key symbols as soon as thefinger is released or decision diamond 792 verifies the impulse haspeaked, ensuring prompt feedback to the user. Fingers intended to reston the surface generate no keys as long as they are members of asynchronized finger press or release subset or are placed on the surfacegently and remain there along with other fingers for a second or two.Once resting, fingers can be lifted and tapped or impulsively pressed onthe surface to generate key symbols without having to lift other restingfingers. Typematic is initiated ether by impulsively pressing andmaintaining distinguishable force on a key, or by holding a finger on akey while other fingers on the hand are lifted. Glancing motions ofsingle fingers as they tap key regions are easily tolerated since mostcursor manipulation must be initiated by synchronized slides of two ormore fingers.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

1. A method for determining which of a plurality of concurrent surfacecontacts of touch devices on or near a touch sensing surface in acurrent proximity image corresponds to a surface contact in a priorproximity image, the method comprising: generating a first predictedpath by predicting a continuation of an existing path of the surfacecontact in the prior proximity image; generating one or more otherpredicted paths of one or more other surface contacts in the priorproximity image; finding a closest surface contact of the plurality ofsurface contacts in the current proximity image, the closest surfacecontact being the surface contact closest to the first predicted path;and pairing the closest surface contact with the first predicted path ifthe first predicted path is closer to the closest surface contact thaneach of the one or more other predicted paths.
 2. A method fordetermining which of a plurality of concurrent surface contacts of touchdevices on or near a touch sensing surface in a current proximity imagecorresponds to a surface contact in a prior proximity image, the methodcomprising: generating a first predicted path by predicting acontinuation of an existing path of the surface contact in the priorproximity image; generating one or more other predicted paths of one ormore other surface contacts in the prior proximity image; finding aclosest surface contact of the plurality of surface contacts in thecurrent proximity image, the closest surface contact being the surfacecontact closest to the first predicted path and within a path-dependenttracking radius of the first predicted path; and pairing the closestsurface contact with the first predicted path if the first predictedpath is closer to the closest surface contact than each of the one ormore other predicted paths, and if the centroid of the closest surfacecontact is within the path-dependent tracking radius of the firstpredicted path.
 3. The method of claim 1 or 2 further comprising:starting new paths for unpaired surface contacts.
 4. The method of claim3 further comprising: deactivating unpaired predicted paths.
 5. Themethod of claim 1 or 2 further comprising: deactivating unpairedpredicted paths.
 6. The method of claim 1 or 2 further comprising:updating a path parameter for the first predicted path from a measuredparameter of the surface contact paired with the first predicted path.7. The method of claim 6 further comprising: starting new paths forunpaired surface contacts.
 8. The method of claim 7 further comprising:deactivating unpaired predicted paths.
 9. The method of claim 6 furthercomprising: deactivating unpaired predicted paths.
 10. The method ofclaim 1 or 2 wherein generating the first predicted path comprisesdetermining a velocity of the surface contact along the existing path.11. The method of claim 10 further comprising: starting new paths forunpaired surface contacts.
 12. The method of claim 11 furthercomprising: deactivating unpaired predicted paths.
 13. The method ofclaim 10 further comprising: deactivating unpaired predicted paths. 14.The method of claim 10 further comprising: updating a path parameter forthe first predicted path from a measured parameter of the surfacecontact paired with the first predicted path.
 15. The method of claim 14further comprising: starting new paths for unpaired surface contacts.16. The method of claim 15 further comprising: deactivating unpairedpredicted paths.
 17. The method of claim 14 further comprising:deactivating unpaired predicted paths.
 18. A method for determiningwhich of a plurality of concurrent groups of pixels in a currentproximity image corresponds to a group of pixels in a prior proximityimage, each group of pixels corresponding to a distinguishable hand partor other touch object on or near a surface of a multi-touch apparatusand each proximity image representing a scan of a plurality of proximitysensors of the multi-touch apparatus, the method comprising: predictinga first predicted path of one of the groups of pixels in the priorproximity image; predicting one or more other predicted paths of one ormore other groups of pixels in the prior proximity image; finding aclosest group of pixels of the plurality of groups of pixels in thecurrent proximity image, the closest group of pixels being the group ofpixels closest to the first predicted path; and pairing the closestgroup of pixels with the first predicted path if the first predictedpath is closer to the closest group of pixels than each of the one ormore other predicted paths, whereby the first predicted path may be usedto generate user interface interactions in response to motion of thedistinguishable hand part or other touch object from the prior proximityimage to the current proximity image.
 19. The method of claim 18 furthercomprising: starting new paths for unpaired groups of pixels.
 20. Themethod of claim 19 further comprising: deactivating unpaired predictedpaths.
 21. The method of claim 18 further comprising: deactivatingunpaired predicted paths.
 22. The method of claim 18 wherein at leastone of the groups of pixels corresponds to a hand part near but not onthe multi-touch surface.
 23. The method of claim 22 further comprising:starting new paths for unpaired groups of pixels, wherein at least oneunpaired group of pixels corresponds to a hand part near but not on themulti-touch surface.